[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:43 dist-test-slave-zkpd krb5kdc[14237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:43 dist-test-slave-zkpd krb5kdc[14237](info): set up 2 sockets
Aug 08 05:59:43 dist-test-slave-zkpd krb5kdc[14237](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 05:59:44.544859 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 05:59:44 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T05:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T05:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 05:59:44.668615 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43887
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:43887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:44.781306 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:44.781602 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:44.781678 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:44.785688 14250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 05:59:44.785780 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:44.785818 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:44.785876 14250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 05:59:44.785916 14250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 05:59:44.791062 14250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:43887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:44.792183 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:44.793233 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:44.802417 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:44.802477 14250 server_base.cc:1034] running on GCE node
W20260808 05:59:44.802429 14255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:44.802388 14259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:44.803122 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:44.804292 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:44.805478 14250 hybrid_clock.cc:648] HybridClock initialized: now 1786168784805428 us; error 63 us; skew 500 ppm
Aug 08 05:59:44 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168784, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:44.808198 14250 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 05:59:44.809036 14250 webserver.cc:460] Webserver started at http://127.13.229.126:44509/ using document root <none> and password file <none>
I20260808 05:59:44.809584 14250 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:44.809672 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:44.809911 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:44.811967 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "814bd694a6cc468096b0cd6e0805309c"
format_stamp: "Formatted at 2026-08-08 05:59:44 on dist-test-slave-zkpd"
server_key: "278471c0c660648906fd1748e666c7ae"
server_key_iv: "ddd4fcb9e5c01ef1347cde8c038a0e0c"
server_key_version: "encryptionkey@0"
I20260808 05:59:44.812534 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "814bd694a6cc468096b0cd6e0805309c"
format_stamp: "Formatted at 2026-08-08 05:59:44 on dist-test-slave-zkpd"
server_key: "278471c0c660648906fd1748e666c7ae"
server_key_iv: "ddd4fcb9e5c01ef1347cde8c038a0e0c"
server_key_version: "encryptionkey@0"
I20260808 05:59:44.816236 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 05:59:44.818534 14266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:44.819723 14250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:44.819882 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "814bd694a6cc468096b0cd6e0805309c"
format_stamp: "Formatted at 2026-08-08 05:59:44 on dist-test-slave-zkpd"
server_key: "278471c0c660648906fd1748e666c7ae"
server_key_iv: "ddd4fcb9e5c01ef1347cde8c038a0e0c"
server_key_version: "encryptionkey@0"
I20260808 05:59:44.820065 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:44.831972 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:44.834713 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:44.834926 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:44.842893 14250 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:43887
I20260808 05:59:44.842954 14322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:43887 every 8 connection(s)
I20260808 05:59:44.843884 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 05:59:44.845376 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14250
I20260808 05:59:44.845541 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 05:59:44.847208 14229 external_mini_cluster.cc:1428] Setting key 0dae5beaec4a4ea32cd73d62cc4ced84
I20260808 05:59:44.847379 14323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:44.853919 14323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: Bootstrap starting.
I20260808 05:59:44.856539 14323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:44.857403 14323 log.cc:824] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:44.859468 14323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: No bootstrap required, opened a new log
I20260808 05:59:44.863579 14323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } }
I20260808 05:59:44.863794 14323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:44.863873 14323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814bd694a6cc468096b0cd6e0805309c, State: Initialized, Role: FOLLOWER
I20260808 05:59:44.864380 14323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } }
I20260808 05:59:44.864507 14323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 05:59:44.864590 14323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 05:59:44.864707 14323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 0 FOLLOWER]: Advancing to term 1
Aug 08 05:59:44 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:44.865751 14323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } }
I20260808 05:59:44.866097 14323 leader_election.cc:302] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814bd694a6cc468096b0cd6e0805309c; no voters:
I20260808 05:59:44.866442 14323 leader_election.cc:288] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 05:59:44.866515 14328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:44.866715 14328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [term 1 LEADER]: Becoming Leader. State: Replica: 814bd694a6cc468096b0cd6e0805309c, State: Running, Role: LEADER
I20260808 05:59:44.867026 14328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } }
I20260808 05:59:44.867707 14323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 05:59:44.868216 14329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814bd694a6cc468096b0cd6e0805309c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } } }
I20260808 05:59:44.868342 14329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:44.869064 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814bd694a6cc468096b0cd6e0805309c. Latest consensus state: current_term: 1 leader_uuid: "814bd694a6cc468096b0cd6e0805309c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814bd694a6cc468096b0cd6e0805309c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43887 } } }
I20260808 05:59:44.869164 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:44.873008 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:44.850800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.13.229.126:43887)
0808 05:59:44.851314 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:44.851343 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:44.852400 (+ 1057us) server_negotiation.cc:407] Connection header received
0808 05:59:44.855733 (+ 3333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:44.855754 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:44.856162 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:44.856518 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:44.858745 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:44.860255 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:44.861386 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:44.862146 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:44.862344 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:44.866320 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:44.866325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:44.866338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:44.866365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:44.868921 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:44.869658 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:44.869662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:44.869666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:44.869772 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:44.870215 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:44.870218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:44.870220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:44.870870 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:44.871054 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:44.871704 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 05:59:44.872013 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":170,"threads_started":1}
W20260808 05:59:44.873353 14347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 05:59:44.873436 14347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 05:59:44.873520 14333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 05:59:44.874282 14333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 05:59:44.880363 14333 catalog_manager.cc:1347] Generated new cluster ID: b597f94c22034f8c8ecb49d051b900be
I20260808 05:59:44.880443 14333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 05:59:44.891858 14333 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 05:59:44.892812 14333 catalog_manager.cc:1504] Loading token signing keys...
I20260808 05:59:44.904736 14333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 814bd694a6cc468096b0cd6e0805309c: Generated new TSK 0
I20260808 05:59:44.905587 14333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 05:59:44.943672 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:45.062879 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:45.063118 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:45.063161 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:45.067230 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:45.067315 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:45.067427 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 05:59:45.072966 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:45.074009 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:45.075003 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:45.084337 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:45.084373 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:45.084394 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:45.084596 14352 server_base.cc:1034] running on GCE node
I20260808 05:59:45.085083 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:45.085682 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:45.086845 14352 hybrid_clock.cc:648] HybridClock initialized: now 1786168785086830 us; error 30 us; skew 500 ppm
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:45.089772 14352 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 05:59:45.090643 14352 webserver.cc:460] Webserver started at http://127.13.229.65:37679/ using document root <none> and password file <none>
I20260808 05:59:45.091156 14352 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:45.091214 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:45.091393 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:45.093508 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "8a4a18d19f23424d9422359beed14f8c"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "63e56982b5b555552348afe85c90883a"
server_key_iv: "b87800e31eb737a92175795b36f7f921"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.094163 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "8a4a18d19f23424d9422359beed14f8c"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "63e56982b5b555552348afe85c90883a"
server_key_iv: "b87800e31eb737a92175795b36f7f921"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.097802 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 05:59:45.100106 14368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.101394 14352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:45.101573 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "8a4a18d19f23424d9422359beed14f8c"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "63e56982b5b555552348afe85c90883a"
server_key_iv: "b87800e31eb737a92175795b36f7f921"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.101708 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:45.114192 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:45.117430 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:45.117579 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:45.118098 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:45.118911 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:45.118960 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.119000 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:45.119016 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.128779 14352 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:37107
I20260808 05:59:45.128829 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:37107 every 8 connection(s)
I20260808 05:59:45.129665 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 05:59:45.130621 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14352
I20260808 05:59:45.130731 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 05:59:45.131027 14229 external_mini_cluster.cc:1428] Setting key 49cf43a89f9f7f7f096285c276baa210
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 05:59:45.145481 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.131738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48267 (local address 127.13.229.126:43887)
0808 05:59:45.131926 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:45.131931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:45.132914 (+ 983us) server_negotiation.cc:407] Connection header received
0808 05:59:45.134937 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:45.134943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:45.135010 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:45.135107 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:45.136382 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.137255 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.138493 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.138962 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.139107 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.141718 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:45.141724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:45.141727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:45.141758 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:45.143385 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.143928 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.143931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.143933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.144019 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.144369 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.144372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.144374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.144559 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:45.144669 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:45.145231 (+ 562us) server_negotiation.cc:299] Negotiation successful
0808 05:59:45.145342 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 05:59:45.146674 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.132197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43887 (local address 127.13.229.65:48267)
0808 05:59:45.132739 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:45.132798 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:45.134668 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:45.135287 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:45.135293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:45.135639 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:45.136214 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.136229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.137382 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.137385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.138357 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.138367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.139123 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.139126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.139267 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.139986 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:45.140002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:45.141519 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:45.143523 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.143529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.143544 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.143798 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.144116 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.144120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.144122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.144257 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.144669 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:45.144676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:45.144966 (+ 290us) client_negotiation.cc:769] Sending connection context
0808 05:59:45.145155 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 05:59:45.145528 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":166,"threads_started":1}
I20260808 05:59:45.147989 14488 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43887
I20260808 05:59:45.148284 14488 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:45.148792 14488 heartbeater.cc:511] Master 127.13.229.126:43887 requested a full tablet report, sending...
I20260808 05:59:45.150527 14283 ts_manager.cc:194] Registered new tserver with Master: 8a4a18d19f23424d9422359beed14f8c (127.13.229.65:37107)
I20260808 05:59:45.152246 14283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:48267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 05:59:45.162595 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:45.277770 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:45.278019 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:45.278061 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:45.282243 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:45.282297 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:45.282356 14493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 05:59:45.287451 14493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:45.288566 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:45.289551 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:45.299305 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:45.299314 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:45.299507 14493 server_base.cc:1034] running on GCE node
W20260808 05:59:45.299314 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:45.300104 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:45.300752 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:45.301944 14493 hybrid_clock.cc:648] HybridClock initialized: now 1786168785301915 us; error 44 us; skew 500 ppm
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:45.304811 14493 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 05:59:45.305712 14493 webserver.cc:460] Webserver started at http://127.13.229.66:45049/ using document root <none> and password file <none>
I20260808 05:59:45.306294 14493 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:45.306389 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:45.306629 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:45.308715 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "f1d0c2b89ad243a391d1ec62b182cfd7"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "50d313fbe5b7b4735ec063587096ac23"
server_key_iv: "8cf29962d7e144cff89bd55b66192756"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.309309 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "f1d0c2b89ad243a391d1ec62b182cfd7"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "50d313fbe5b7b4735ec063587096ac23"
server_key_iv: "8cf29962d7e144cff89bd55b66192756"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.312991 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 05:59:45.315291 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.316655 14493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:45.316792 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "f1d0c2b89ad243a391d1ec62b182cfd7"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "50d313fbe5b7b4735ec063587096ac23"
server_key_iv: "8cf29962d7e144cff89bd55b66192756"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.316902 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:45.329784 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:45.333103 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:45.333307 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:45.333918 14493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:45.334729 14493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:45.334815 14493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.334888 14493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:45.334932 14493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.344264 14493 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:38855
I20260808 05:59:45.344285 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:38855 every 8 connection(s)
I20260808 05:59:45.345152 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 05:59:45.349306 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14493
I20260808 05:59:45.349442 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 05:59:45.349694 14229 external_mini_cluster.cc:1428] Setting key 7af939d1cf9d9e5974ea49725abc8609
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:45.360462 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.347022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:48573 (local address 127.13.229.126:43887)
0808 05:59:45.347175 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:45.347179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:45.348049 (+ 870us) server_negotiation.cc:407] Connection header received
0808 05:59:45.349346 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:45.349350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:45.349409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:45.349519 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:45.350963 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.351711 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.352947 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.353351 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.353459 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.356272 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:45.356278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:45.356282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:45.356313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:45.358046 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.358633 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.358637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.358640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.358756 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.359102 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.359106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.359108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.359362 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:45.359481 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:45.360127 (+ 646us) server_negotiation.cc:299] Negotiation successful
0808 05:59:45.360267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 05:59:45.361332 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.347327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43887 (local address 127.13.229.66:48573)
0808 05:59:45.347840 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:45.347935 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:45.349126 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:45.349690 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:45.349697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:45.350140 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:45.350793 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.350813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.351882 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.351885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.352822 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.352831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.353525 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.353530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.353778 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.354421 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:45.354447 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:45.356100 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:45.358164 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.358169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.358180 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.358510 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.358854 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.358857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.358858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.358990 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.359487 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:45.359493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:45.359811 (+ 318us) client_negotiation.cc:769] Sending connection context
0808 05:59:45.360039 (+ 228us) client_negotiation.cc:240] Negotiation successful
0808 05:59:45.360429 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":105,"threads_started":1}
I20260808 05:59:45.362443 14629 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43887
I20260808 05:59:45.362684 14629 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:45.363260 14629 heartbeater.cc:511] Master 127.13.229.126:43887 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 05:59:45.364486 14283 ts_manager.cc:194] Registered new tserver with Master: f1d0c2b89ad243a391d1ec62b182cfd7 (127.13.229.66:38855)
I20260808 05:59:45.365047 14283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:48573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 05:59:45.379474 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:45.498099 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:45.498334 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:45.498412 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:45.502425 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:45.502490 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:45.502599 14634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 05:59:45.507783 14634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:45.508790 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:45.509776 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:45.519119 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:45.519119 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:45.519172 14634 server_base.cc:1034] running on GCE node
W20260808 05:59:45.519255 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:45.519734 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:45.520387 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:45.521577 14634 hybrid_clock.cc:648] HybridClock initialized: now 1786168785521555 us; error 39 us; skew 500 ppm
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:45.524798 14634 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 05:59:45.525796 14634 webserver.cc:460] Webserver started at http://127.13.229.67:34011/ using document root <none> and password file <none>
I20260808 05:59:45.526288 14634 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:45.526345 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:45.526506 14634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:45.528491 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "f253c54e2c1f46b29c22a1a21bd294cc"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "9a658b28f6da44418b1d221d8fca3555"
server_key_iv: "b873e2887da724c10406482c60286aeb"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.528991 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "f253c54e2c1f46b29c22a1a21bd294cc"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "9a658b28f6da44418b1d221d8fca3555"
server_key_iv: "b873e2887da724c10406482c60286aeb"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.532541 14634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 05:59:45.534749 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.536052 14634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:45.536167 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "f253c54e2c1f46b29c22a1a21bd294cc"
format_stamp: "Formatted at 2026-08-08 05:59:45 on dist-test-slave-zkpd"
server_key: "9a658b28f6da44418b1d221d8fca3555"
server_key_iv: "b873e2887da724c10406482c60286aeb"
server_key_version: "encryptionkey@0"
I20260808 05:59:45.536262 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:45.553673 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:45.556597 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:45.556813 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:45.557377 14634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:45.558171 14634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:45.558216 14634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.558281 14634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:45.558319 14634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:45.568115 14634 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:40589
I20260808 05:59:45.568202 14771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:40589 every 8 connection(s)
I20260808 05:59:45.569144 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 05:59:45.575594 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14634
I20260808 05:59:45.575733 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 05:59:45.576071 14229 external_mini_cluster.cc:1428] Setting key b04fa102dcf06e6ba1370837a5e01f7f
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:45.582707 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.570962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:50769 (local address 127.13.229.126:43887)
0808 05:59:45.571098 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:45.571102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:45.572433 (+ 1331us) server_negotiation.cc:407] Connection header received
0808 05:59:45.573343 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:45.573347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:45.573405 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:45.573517 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:45.574760 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.575300 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.576678 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.576955 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.577064 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.579530 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:45.579533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:45.579536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:45.579559 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:45.580897 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.581348 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.581350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.581351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.581426 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.581768 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.581772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.581773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.581964 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:45.582053 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:45.582491 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 05:59:45.582591 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 05:59:45.583477 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.571299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43887 (local address 127.13.229.67:50769)
0808 05:59:45.572283 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:45.572327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:45.573135 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:45.573693 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:45.573700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:45.574079 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:45.574609 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.574625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.575448 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.575452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.576546 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:45.576557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.577113 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.577117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:45.577291 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.577889 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:45.577905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:45.579380 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:45.581001 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.581006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.581017 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.581263 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.581504 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:45.581506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:45.581508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:45.581633 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:45.582062 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:45.582066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:45.582282 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 05:59:45.582435 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 05:59:45.582667 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":598,"threads_started":1}
I20260808 05:59:45.584614 14772 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43887
I20260808 05:59:45.584851 14772 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:45.585336 14772 heartbeater.cc:511] Master 127.13.229.126:43887 requested a full tablet report, sending...
I20260808 05:59:45.586362 14283 ts_manager.cc:194] Registered new tserver with Master: f253c54e2c1f46b29c22a1a21bd294cc (127.13.229.67:40589)
I20260808 05:59:45.586938 14283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:50769
I20260808 05:59:45.590550 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 05:59:45.617761 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.608264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.13.229.65:37107)
0808 05:59:45.608667 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:45.608682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:45.608705 (+ 23us) server_negotiation.cc:407] Connection header received
0808 05:59:45.608758 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:45.608764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:45.608988 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:45.609116 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:45.609770 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.610583 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.611379 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.611735 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.611852 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.614238 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:45.614247 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:45.614260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:45.614286 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:45.616131 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.616521 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.616525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.616527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.616603 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.616841 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.616843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.616844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.617076 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:45.617228 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:45.617385 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 05:59:45.617549 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":165,"threads_started":1}
W20260808 05:59:45.618816 14396 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34572
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 05:59:45.646440 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:45.637996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.13.229.65:37107)
0808 05:59:45.638167 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:45.638172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:45.638308 (+ 136us) server_negotiation.cc:407] Connection header received
0808 05:59:45.638509 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:45.638512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:45.638555 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:45.638649 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:45.639347 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.639877 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.640753 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:45.640995 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:45.641107 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:45.643451 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:45.643455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:45.643457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:45.643480 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:45.644786 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.645215 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.645219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.645221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.645301 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:45.645572 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:45.645576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:45.645578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:45.645761 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:45.645905 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:45.646148 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 05:59:45.646269 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 05:59:45.647521 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14352
I20260808 05:59:45.655704 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14493
I20260808 05:59:45.662221 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14634
I20260808 05:59:45.668502 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14250
2026-08-08T05:59:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2211 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14786](info): set up 2 sockets
Aug 08 05:59:45 dist-test-slave-zkpd krb5kdc[14786](info): commencing operation
krb5kdc: starting...
W20260808 05:59:46.740656 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 05:59:46 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T05:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T05:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 05:59:46.864605 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33371
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:46.975251 14799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:46.975493 14799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:46.975530 14799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:46.979350 14799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 05:59:46.979406 14799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:46.979425 14799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:46.979441 14799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 05:59:46.979456 14799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 05:59:46.984643 14799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:46.985642 14799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:46.986654 14799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:46.994932 14805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:46.994925 14804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:46.995128 14799 server_base.cc:1034] running on GCE node
W20260808 05:59:46.994927 14807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:46.995798 14799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:46.996882 14799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:46.998047 14799 hybrid_clock.cc:648] HybridClock initialized: now 1786168786998018 us; error 39 us; skew 500 ppm
Aug 08 05:59:46 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168786, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:47.000803 14799 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 05:59:47.001677 14799 webserver.cc:460] Webserver started at http://127.13.229.126:40233/ using document root <none> and password file <none>
I20260808 05:59:47.002221 14799 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:47.002315 14799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:47.002542 14799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:47.004638 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "96b33d30f38848a7b803222180013fdf"
format_stamp: "Formatted at 2026-08-08 05:59:46 on dist-test-slave-zkpd"
server_key: "2ac57f824cf115b237675006a9cf9b44"
server_key_iv: "a1aa3ea72a02ad056f388ae106c42fc5"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.005218 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "96b33d30f38848a7b803222180013fdf"
format_stamp: "Formatted at 2026-08-08 05:59:46 on dist-test-slave-zkpd"
server_key: "2ac57f824cf115b237675006a9cf9b44"
server_key_iv: "a1aa3ea72a02ad056f388ae106c42fc5"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.008838 14799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 05:59:47.011140 14814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.012413 14799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:47.012552 14799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "96b33d30f38848a7b803222180013fdf"
format_stamp: "Formatted at 2026-08-08 05:59:46 on dist-test-slave-zkpd"
server_key: "2ac57f824cf115b237675006a9cf9b44"
server_key_iv: "a1aa3ea72a02ad056f388ae106c42fc5"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.012658 14799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:47.025808 14799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:47.028584 14799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:47.028789 14799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:47.035804 14799 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:33371
I20260808 05:59:47.035823 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:33371 every 8 connection(s)
I20260808 05:59:47.036757 14799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 05:59:47.039395 14874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:47.040578 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14799
I20260808 05:59:47.040695 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 05:59:47.040988 14229 external_mini_cluster.cc:1428] Setting key 00ef55a866db3f981d4d7a2c83e5b16e
I20260808 05:59:47.045478 14874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf: Bootstrap starting.
I20260808 05:59:47.047674 14874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:47.048542 14874 log.cc:824] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf: Log is configured to *not* fsync() on all Append() calls
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:47.050681 14874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf: No bootstrap required, opened a new log
I20260808 05:59:47.054093 14874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } }
I20260808 05:59:47.054342 14874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:47.054420 14874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b33d30f38848a7b803222180013fdf, State: Initialized, Role: FOLLOWER
I20260808 05:59:47.054877 14874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } }
I20260808 05:59:47.055047 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.042411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.13.229.126:33371)
0808 05:59:47.042868 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.042883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.042918 (+ 35us) server_negotiation.cc:407] Connection header received
0808 05:59:47.043529 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.043544 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.043860 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.044243 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.045038 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.046093 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.046947 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.047413 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.047590 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.050276 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.050290 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.050306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.050337 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.052138 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.052566 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.052572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.052576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.052677 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.052956 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.052960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.052961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.053374 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.053582 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.053919 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.054181 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260808 05:59:47.055017 14874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 05:59:47.055174 14874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 05:59:47.055291 14874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:47.056310 14874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } }
I20260808 05:59:47.056660 14874 leader_election.cc:302] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b33d30f38848a7b803222180013fdf; no voters:
I20260808 05:59:47.056973 14874 leader_election.cc:288] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 05:59:47.057073 14879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:47.057291 14879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [term 1 LEADER]: Becoming Leader. State: Replica: 96b33d30f38848a7b803222180013fdf, State: Running, Role: LEADER
I20260808 05:59:47.057600 14879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } }
I20260808 05:59:47.058140 14874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 05:59:47.059149 14880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b33d30f38848a7b803222180013fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } } }
I20260808 05:59:47.059294 14880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:47.059645 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b33d30f38848a7b803222180013fdf. Latest consensus state: current_term: 1 leader_uuid: "96b33d30f38848a7b803222180013fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b33d30f38848a7b803222180013fdf" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33371 } } }
I20260808 05:59:47.059710 14888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 05:59:47.059743 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:47.062764 14888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 05:59:47.068094 14888 catalog_manager.cc:1347] Generated new cluster ID: 0294b19232c546ab8f654d091afbdaa4
I20260808 05:59:47.068156 14888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 05:59:47.073328 14888 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 05:59:47.074544 14888 catalog_manager.cc:1504] Loading token signing keys...
I20260808 05:59:47.083302 14888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b33d30f38848a7b803222180013fdf: Generated new TSK 0
I20260808 05:59:47.084017 14888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 05:59:47.113029 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:47.225049 14902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:47.225291 14902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:47.225334 14902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:47.229292 14902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:47.229346 14902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:47.229409 14902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 05:59:47.234416 14902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:47.235383 14902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:47.236341 14902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:47.245894 14907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:47.245890 14910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:47.245965 14908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:47.246420 14902 server_base.cc:1034] running on GCE node
I20260808 05:59:47.246833 14902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:47.247529 14902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:47.248725 14902 hybrid_clock.cc:648] HybridClock initialized: now 1786168787248697 us; error 37 us; skew 500 ppm
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:47.251506 14902 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 05:59:47.252425 14902 webserver.cc:460] Webserver started at http://127.13.229.65:44293/ using document root <none> and password file <none>
I20260808 05:59:47.253016 14902 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:47.253080 14902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:47.253352 14902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:47.256017 14902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "e5eadfb5892e4783a4574469e6c9ab0d"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "347dad353930471eab5151972b6c5026"
server_key_iv: "48ff66ad97d1e78346586f85a50993f4"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.256752 14902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "e5eadfb5892e4783a4574469e6c9ab0d"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "347dad353930471eab5151972b6c5026"
server_key_iv: "48ff66ad97d1e78346586f85a50993f4"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.260703 14902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 05:59:47.263031 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.264297 14902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:47.264427 14902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "e5eadfb5892e4783a4574469e6c9ab0d"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "347dad353930471eab5151972b6c5026"
server_key_iv: "48ff66ad97d1e78346586f85a50993f4"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.264533 14902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:47.273483 14902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:47.276182 14902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:47.276372 14902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:47.276927 14902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:47.277709 14902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:47.277786 14902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.277873 14902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:47.277922 14902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.286732 14902 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:33445
I20260808 05:59:47.286764 15036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:33445 every 8 connection(s)
I20260808 05:59:47.287580 14902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 05:59:47.289404 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 14902
I20260808 05:59:47.289525 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 05:59:47.289779 14229 external_mini_cluster.cc:1428] Setting key 1e57871f131a6d34817b7bbd01467a0c
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 05:59:47.302852 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.289402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:57365 (local address 127.13.229.126:33371)
0808 05:59:47.289573 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.289577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.290438 (+ 861us) server_negotiation.cc:407] Connection header received
0808 05:59:47.292121 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.292123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.292165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.292235 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.293797 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.294652 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.295773 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.296086 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.296202 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.299225 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.299231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.299238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.299271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.300939 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.301414 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.301416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.301417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.301490 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.301769 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.301771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.301772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.301948 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.302027 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.302533 (+ 506us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.302710 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 05:59:47.303751 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.289747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33371 (local address 127.13.229.65:57365)
0808 05:59:47.290260 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.290306 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.291846 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.292414 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.292426 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.292960 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:47.293635 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.293652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.294796 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.294799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.295639 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.295649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.296241 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.296245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.296418 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.297035 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:47.297051 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:47.299032 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:47.301054 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.301058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.301070 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.301317 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.301570 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.301573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.301574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.301698 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.302072 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:47.302077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:47.302320 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.302499 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.302770 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260808 05:59:47.305029 15037 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33371
I20260808 05:59:47.305306 15037 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:47.305812 15037 heartbeater.cc:511] Master 127.13.229.126:33371 requested a full tablet report, sending...
I20260808 05:59:47.307737 14833 ts_manager.cc:194] Registered new tserver with Master: e5eadfb5892e4783a4574469e6c9ab0d (127.13.229.65:33445)
I20260808 05:59:47.309001 14833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:57365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 05:59:47.322454 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:47.434871 15042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:47.435118 15042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:47.435185 15042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:47.439265 15042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:47.439318 15042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:47.439430 15042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 05:59:47.444928 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:47.445922 15042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:47.446899 15042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:47.456347 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:47.456334 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:47.456378 15042 server_base.cc:1034] running on GCE node
W20260808 05:59:47.456334 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:47.456929 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:47.457559 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:47.458791 15042 hybrid_clock.cc:648] HybridClock initialized: now 1786168787458749 us; error 47 us; skew 500 ppm
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:47.462680 15042 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 05:59:47.463712 15042 webserver.cc:460] Webserver started at http://127.13.229.66:41095/ using document root <none> and password file <none>
I20260808 05:59:47.464572 15042 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:47.464658 15042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:47.464911 15042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:47.467440 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "22f1984f47044d798530c14ff029c285"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "40d584537ce3cc2c4620926cd88cfe8d"
server_key_iv: "e0f0e93c70c2848dc186fbab8e401a29"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.468236 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "22f1984f47044d798530c14ff029c285"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "40d584537ce3cc2c4620926cd88cfe8d"
server_key_iv: "e0f0e93c70c2848dc186fbab8e401a29"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.472270 15042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 05:59:47.474601 15059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.475746 15042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:47.475883 15042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "22f1984f47044d798530c14ff029c285"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "40d584537ce3cc2c4620926cd88cfe8d"
server_key_iv: "e0f0e93c70c2848dc186fbab8e401a29"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.476053 15042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:47.489117 15042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:47.492224 15042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:47.492431 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:47.493067 15042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:47.493947 15042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:47.494021 15042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.494098 15042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:47.494151 15042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.503849 15042 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:32855
I20260808 05:59:47.503875 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:32855 every 8 connection(s)
I20260808 05:59:47.504813 15042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 05:59:47.509516 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15042
I20260808 05:59:47.509658 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 05:59:47.509917 14229 external_mini_cluster.cc:1428] Setting key 6affae7956c9e6066c0ab846f2a6d4a7
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:47.520865 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.506727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:50651 (local address 127.13.229.126:33371)
0808 05:59:47.506988 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.506991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.507719 (+ 728us) server_negotiation.cc:407] Connection header received
0808 05:59:47.509126 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.509130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.509188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.509297 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.510673 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.511461 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.513251 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.513512 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.513651 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.516433 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.516436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.516439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.516470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.518325 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.518914 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.518940 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.518943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.519045 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.519391 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.519394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.519396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.519655 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.519768 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.520482 (+ 714us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.520625 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 05:59:47.521590 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.507108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33371 (local address 127.13.229.66:50651)
0808 05:59:47.507573 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.507615 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.508916 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.509481 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.509488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.509939 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:47.510512 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.510532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.511602 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.511607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.513101 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.513121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.513696 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.513707 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.513895 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.514511 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:47.514527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:47.516261 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:47.518454 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.518459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.518470 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.518759 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.519132 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.519134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.519135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.519265 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.519769 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:47.519775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:47.520148 (+ 373us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.520362 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.520622 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260808 05:59:47.522858 15179 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33371
I20260808 05:59:47.523073 15179 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:47.523521 15179 heartbeater.cc:511] Master 127.13.229.126:33371 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 05:59:47.524763 14833 ts_manager.cc:194] Registered new tserver with Master: 22f1984f47044d798530c14ff029c285 (127.13.229.66:32855)
I20260808 05:59:47.525368 14833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:50651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 05:59:47.543035 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:47.666181 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:47.666465 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:47.666546 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:47.670746 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:47.670838 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:47.670933 15185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 05:59:47.676342 15185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33371
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:47.677429 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:47.678480 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:47.688169 15191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:47.688230 15185 server_base.cc:1034] running on GCE node
W20260808 05:59:47.688150 15192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:47.688169 15194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:47.688846 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:47.689422 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:47.690590 15185 hybrid_clock.cc:648] HybridClock initialized: now 1786168787690570 us; error 29 us; skew 500 ppm
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:47.693547 15185 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 05:59:47.694439 15185 webserver.cc:460] Webserver started at http://127.13.229.67:35781/ using document root <none> and password file <none>
I20260808 05:59:47.694960 15185 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:47.695019 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:47.695181 15185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:47.697211 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "eb828c95c60f444aa09d6f3a705bf675"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "2151997ef6c5c9fa57fc1376e03b01f5"
server_key_iv: "8a72621541e13bcbad2f916e330d5c79"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.697731 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "eb828c95c60f444aa09d6f3a705bf675"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "2151997ef6c5c9fa57fc1376e03b01f5"
server_key_iv: "8a72621541e13bcbad2f916e330d5c79"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.701376 15185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 05:59:47.703665 15201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.704921 15185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:47.705040 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "eb828c95c60f444aa09d6f3a705bf675"
format_stamp: "Formatted at 2026-08-08 05:59:47 on dist-test-slave-zkpd"
server_key: "2151997ef6c5c9fa57fc1376e03b01f5"
server_key_iv: "8a72621541e13bcbad2f916e330d5c79"
server_key_version: "encryptionkey@0"
I20260808 05:59:47.705145 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:47.722157 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:47.725047 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:47.725250 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:47.725876 15185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:47.726706 15185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:47.726754 15185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.726794 15185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:47.726847 15185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:47.736356 15185 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:37823
I20260808 05:59:47.736393 15321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:37823 every 8 connection(s)
I20260808 05:59:47.737231 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 05:59:47.740432 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15185
I20260808 05:59:47.740523 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 05:59:47.740815 14229 external_mini_cluster.cc:1428] Setting key 0b7bb354dcefe3d07dd6395cca112bdf
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:47.751081 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.738998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:52959 (local address 127.13.229.126:33371)
0808 05:59:47.739136 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.739140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.740045 (+ 905us) server_negotiation.cc:407] Connection header received
0808 05:59:47.741567 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.741569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.741644 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.741731 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.743308 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.743849 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.745034 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.745245 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.745325 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.747845 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.747848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.747850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.747939 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.749217 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.749708 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.749710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.749711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.749784 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.750100 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.750102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.750103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.750292 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.750392 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.750849 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.750957 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 05:59:47.752014 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.739361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33371 (local address 127.13.229.67:52959)
0808 05:59:47.739832 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.739877 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.741321 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.741930 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.741937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.742537 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:47.743172 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.743188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.744027 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.744030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.744923 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.744933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.745378 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.745381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.745520 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.746065 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:47.746081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:47.747679 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:47.749332 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.749338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.749350 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.749620 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.749865 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:47.749867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:47.749868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:47.750014 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:47.750403 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:47.750408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:47.750641 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.750809 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.751070 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260808 05:59:47.753229 15322 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33371
I20260808 05:59:47.753517 15322 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:47.754114 15322 heartbeater.cc:511] Master 127.13.229.126:33371 requested a full tablet report, sending...
I20260808 05:59:47.755190 14833 ts_manager.cc:194] Registered new tserver with Master: eb828c95c60f444aa09d6f3a705bf675 (127.13.229.67:37823)
I20260808 05:59:47.755775 14833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:52959
I20260808 05:59:47.766731 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 05:59:47.777621 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.770304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.13.229.126:33371)
0808 05:59:47.770432 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.770435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.770591 (+ 156us) server_negotiation.cc:407] Connection header received
0808 05:59:47.770689 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.770691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.770730 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.770797 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.771538 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.772094 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.772900 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.773099 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.773173 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.774534 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.774536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.774538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.774556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.775962 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.776458 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.776460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.776462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.776540 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.776785 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.776787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.776788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.777010 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.777111 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.777317 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.777409 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 05:59:47.782676 14833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 05:59:47.785763 14833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:47.800034 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.794204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:32855 (local address 127.0.0.1:50340)
0808 05:59:47.794824 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.794874 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.795008 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.796080 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.796083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.796098 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:47.796189 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.796195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797233 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.797237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.798532 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.798543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.799481 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.799485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.799592 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.799628 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.799733 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.799857 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":108,"threads_started":1}
I20260808 05:59:47.800045 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.794330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.13.229.66:32855)
0808 05:59:47.795638 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.795652 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.795670 (+ 18us) server_negotiation.cc:407] Connection header received
0808 05:59:47.795734 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.795739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.795910 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.796097 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:47.796287 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797091 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.798645 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.799349 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.799483 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.799633 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.799779 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.799875 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":96,"threads_started":1}
I20260808 05:59:47.800530 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.794301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:37823 (local address 127.0.0.1:33940)
0808 05:59:47.794906 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.794920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.795008 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.795950 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.795956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.795981 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:47.796109 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.796115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797569 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.797572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.798682 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.798690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.800107 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.800110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.800195 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.800217 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.800297 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.800377 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":2,"thread_start_us":131,"threads_started":1}
I20260808 05:59:47.800592 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.795016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.13.229.67:37823)
0808 05:59:47.795331 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.795347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.795373 (+ 26us) server_negotiation.cc:407] Connection header received
0808 05:59:47.795452 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.795458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.795663 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.795839 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:47.796241 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797431 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.798801 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.799719 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.799851 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.800041 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.800344 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.800452 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20260808 05:59:47.800915 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.795026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.13.229.65:33445)
0808 05:59:47.795419 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.795437 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.795624 (+ 187us) server_negotiation.cc:407] Connection header received
0808 05:59:47.795707 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.795712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.795943 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.796081 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:47.796514 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797626 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.799030 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.799997 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.800100 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.800220 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.800599 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.800709 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260808 05:59:47.801163 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.794627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:33445 (local address 127.0.0.1:47568)
0808 05:59:47.795401 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.795415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.795530 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.796256 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.796259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.796280 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:47.796381 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.796388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.797752 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.797757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.798901 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.798912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.800281 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.800288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.800383 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.800403 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.800470 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.800524 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":85,"threads_started":1}
I20260808 05:59:47.803371 15251 tablet_service.cc:1467] Processing CreateTablet for tablet a974f4109de94e1d866674c5e9a704fd (DEFAULT_TABLE table=test-table [id=231cc31e9c004a31af155e8284856c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:47.803460 15107 tablet_service.cc:1467] Processing CreateTablet for tablet a974f4109de94e1d866674c5e9a704fd (DEFAULT_TABLE table=test-table [id=231cc31e9c004a31af155e8284856c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:47.803861 14966 tablet_service.cc:1467] Processing CreateTablet for tablet a974f4109de94e1d866674c5e9a704fd (DEFAULT_TABLE table=test-table [id=231cc31e9c004a31af155e8284856c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:47.804476 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a974f4109de94e1d866674c5e9a704fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:47.804718 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a974f4109de94e1d866674c5e9a704fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:47.804790 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a974f4109de94e1d866674c5e9a704fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:47.811180 15342 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Bootstrap starting.
I20260808 05:59:47.812278 15343 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: Bootstrap starting.
I20260808 05:59:47.812601 15344 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Bootstrap starting.
I20260808 05:59:47.813635 15342 tablet_bootstrap.cc:654] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:47.814504 15343 tablet_bootstrap.cc:654] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:47.814754 15344 tablet_bootstrap.cc:654] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:47.815056 15342 log.cc:824] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:47.815342 15343 log.cc:824] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:47.815577 15344 log.cc:824] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:47.817483 15343 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: No bootstrap required, opened a new log
I20260808 05:59:47.817631 15342 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: No bootstrap required, opened a new log
I20260808 05:59:47.817713 15343 ts_tablet_manager.cc:1397] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 05:59:47.817829 15342 ts_tablet_manager.cc:1397] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 05:59:47.817947 15344 tablet_bootstrap.cc:492] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: No bootstrap required, opened a new log
I20260808 05:59:47.818120 15344 ts_tablet_manager.cc:1397] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260808 05:59:47.821352 15343 raft_consensus.cc:348] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.821600 15343 raft_consensus.cc:374] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:47.821655 15343 raft_consensus.cc:729] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5eadfb5892e4783a4574469e6c9ab0d, State: Initialized, Role: FOLLOWER
I20260808 05:59:47.821875 15344 raft_consensus.cc:348] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.821875 15342 raft_consensus.cc:348] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.822142 15342 raft_consensus.cc:374] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:47.822142 15344 raft_consensus.cc:374] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:47.822202 15344 raft_consensus.cc:729] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f1984f47044d798530c14ff029c285, State: Initialized, Role: FOLLOWER
I20260808 05:59:47.822202 15342 raft_consensus.cc:729] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb828c95c60f444aa09d6f3a705bf675, State: Initialized, Role: FOLLOWER
I20260808 05:59:47.822113 15343 consensus_queue.cc:260] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.822633 15342 consensus_queue.cc:260] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.822633 15344 consensus_queue.cc:260] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.823204 15343 ts_tablet_manager.cc:1428] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 05:59:47.823550 15037 heartbeater.cc:503] Master 127.13.229.126:33371 was elected leader, sending a full tablet report...
I20260808 05:59:47.823580 15344 ts_tablet_manager.cc:1428] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260808 05:59:47.823580 15342 ts_tablet_manager.cc:1428] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 05:59:47.823928 15179 heartbeater.cc:503] Master 127.13.229.126:33371 was elected leader, sending a full tablet report...
I20260808 05:59:47.823925 15322 heartbeater.cc:503] Master 127.13.229.126:33371 was elected leader, sending a full tablet report...
I20260808 05:59:47.901129 15352 raft_consensus.cc:482] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:47.901331 15352 raft_consensus.cc:504] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.902535 15352 leader_election.cc:288] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb828c95c60f444aa09d6f3a705bf675 (127.13.229.67:37823), e5eadfb5892e4783a4574469e6c9ab0d (127.13.229.65:33445)
I20260808 05:59:47.907337 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.902897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:49911 (local address 127.13.229.67:37823)
0808 05:59:47.903024 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.903030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.903045 (+ 15us) server_negotiation.cc:407] Connection header received
0808 05:59:47.903197 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.903200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.903245 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.903311 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:47.903660 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.904478 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906076 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.906679 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906809 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.906874 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.907133 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.907204 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 05:59:47.907337 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.902967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:41067 (local address 127.13.229.65:33445)
0808 05:59:47.903096 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.903102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.903388 (+ 286us) server_negotiation.cc:407] Connection header received
0808 05:59:47.903505 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.903507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.903549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.903642 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:47.903883 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.904639 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906076 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.906686 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906812 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.906865 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.907133 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.907204 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 05:59:47.907353 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.902759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:37823 (local address 127.13.229.66:49911)
0808 05:59:47.902959 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.902974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.903074 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.903354 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.903358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.903397 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:47.903545 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.903550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.904599 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.904601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.905942 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.905954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.906808 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.907000 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.907019 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.907093 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.907191 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":114,"spinlock_wait_cycles":29184}
I20260808 05:59:47.907361 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.902818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:33445 (local address 127.13.229.66:41067)
0808 05:59:47.903249 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:47.903260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:47.903404 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:47.903651 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:47.903654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:47.903680 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:47.903777 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.903782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.904736 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.904739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.905942 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:47.905954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.906817 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.906819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:47.906946 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.907012 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 05:59:47.907093 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 05:59:47.907192 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":49,"spinlock_wait_cycles":8064,"thread_start_us":138,"threads_started":1}
I20260808 05:59:47.908180 15272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a974f4109de94e1d866674c5e9a704fd" candidate_uuid: "22f1984f47044d798530c14ff029c285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb828c95c60f444aa09d6f3a705bf675" is_pre_election: true
I20260808 05:59:47.908407 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a974f4109de94e1d866674c5e9a704fd" candidate_uuid: "22f1984f47044d798530c14ff029c285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" is_pre_election: true
I20260808 05:59:47.908547 15272 raft_consensus.cc:2454] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f1984f47044d798530c14ff029c285 in term 0.
I20260808 05:59:47.908715 14988 raft_consensus.cc:2454] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f1984f47044d798530c14ff029c285 in term 0.
I20260808 05:59:47.909242 15061 leader_election.cc:302] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22f1984f47044d798530c14ff029c285, e5eadfb5892e4783a4574469e6c9ab0d, eb828c95c60f444aa09d6f3a705bf675; no voters:
I20260808 05:59:47.909554 15352 raft_consensus.cc:2781] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:47.909650 15352 raft_consensus.cc:482] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:47.909714 15352 raft_consensus.cc:3037] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:47.910864 15352 raft_consensus.cc:504] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.911237 15352 leader_election.cc:288] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [CANDIDATE]: Term 1 election: Requested vote from peers eb828c95c60f444aa09d6f3a705bf675 (127.13.229.67:37823), e5eadfb5892e4783a4574469e6c9ab0d (127.13.229.65:33445)
I20260808 05:59:47.911597 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a974f4109de94e1d866674c5e9a704fd" candidate_uuid: "22f1984f47044d798530c14ff029c285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5eadfb5892e4783a4574469e6c9ab0d"
I20260808 05:59:47.911651 15272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a974f4109de94e1d866674c5e9a704fd" candidate_uuid: "22f1984f47044d798530c14ff029c285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb828c95c60f444aa09d6f3a705bf675"
I20260808 05:59:47.911733 14988 raft_consensus.cc:3037] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:47.911783 15272 raft_consensus.cc:3037] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:47.913105 14988 raft_consensus.cc:2454] T a974f4109de94e1d866674c5e9a704fd P e5eadfb5892e4783a4574469e6c9ab0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f1984f47044d798530c14ff029c285 in term 1.
I20260808 05:59:47.913156 15272 raft_consensus.cc:2454] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f1984f47044d798530c14ff029c285 in term 1.
I20260808 05:59:47.913492 15062 leader_election.cc:302] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f1984f47044d798530c14ff029c285, e5eadfb5892e4783a4574469e6c9ab0d; no voters:
I20260808 05:59:47.913756 15352 raft_consensus.cc:2781] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:47.914032 15352 raft_consensus.cc:686] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [term 1 LEADER]: Becoming Leader. State: Replica: 22f1984f47044d798530c14ff029c285, State: Running, Role: LEADER
I20260808 05:59:47.914371 15352 consensus_queue.cc:237] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } }
I20260808 05:59:47.917639 14831 catalog_manager.cc:5495] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22f1984f47044d798530c14ff029c285 (127.13.229.66). New cstate: current_term: 1 leader_uuid: "22f1984f47044d798530c14ff029c285" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f1984f47044d798530c14ff029c285" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 32855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb828c95c60f444aa09d6f3a705bf675" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 37823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5eadfb5892e4783a4574469e6c9ab0d" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 33445 } health_report { overall_health: UNKNOWN } } }
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 05:59:47.954044 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.945134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.13.229.65:33445)
0808 05:59:47.945285 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.945290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.945493 (+ 203us) server_negotiation.cc:407] Connection header received
0808 05:59:47.945599 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.945601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.945644 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.945713 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.946418 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.946904 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.947736 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.948005 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.948088 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.950769 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.950782 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.950798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.950821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.952402 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.952803 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.952808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.952809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.952886 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.953123 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.953125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.953126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.953358 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.953531 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.953715 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.953824 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 05:59:47.982254 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:47.974077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.13.229.65:33445)
0808 05:59:47.974227 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:47.974233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:47.974272 (+ 39us) server_negotiation.cc:407] Connection header received
0808 05:59:47.974414 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:47.974417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:47.974457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:47.974526 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:47.975138 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.975648 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.976496 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:47.976720 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:47.976801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:47.979134 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:47.979138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:47.979141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:47.979176 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:47.980586 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.981008 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.981013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.981015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.981094 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:47.981381 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:47.981384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:47.981386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:47.981550 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:47.981688 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:47.981967 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 05:59:47.982087 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 05:59:47.988178 15323 tablet.cc:2369] T a974f4109de94e1d866674c5e9a704fd P eb828c95c60f444aa09d6f3a705bf675: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 05:59:47 dist-test-slave-zkpd krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 05:59:48 dist-test-slave-zkpd krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
W20260808 05:59:48.006019 15180 tablet.cc:2369] T a974f4109de94e1d866674c5e9a704fd P 22f1984f47044d798530c14ff029c285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 05:59:48.008927 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:48.000122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.13.229.65:33445)
0808 05:59:48.000294 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:48.000299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:48.000449 (+ 150us) server_negotiation.cc:407] Connection header received
0808 05:59:48.000624 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:48.000627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:48.000667 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:48.000751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:48.001435 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:48.001943 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:48.002818 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:48.003037 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:48.003146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:48.005557 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:48.005566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:48.005568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:48.005591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:48.007135 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:48.007597 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:48.007600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:48.007602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:48.007683 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:48.007996 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:48.008000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:48.008002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:48.008216 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:48.008361 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:48.008605 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 05:59:48.008728 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 05:59:48.010262 14946 tablet_service.cc:2908] Rejecting scan request for tablet a974f4109de94e1d866674c5e9a704fd: Uninitialized: clean time has not yet been initialized
I20260808 05:59:48.012300 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14902
I20260808 05:59:48.020138 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15042
I20260808 05:59:48.027863 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15185
I20260808 05:59:48.034245 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 14799
2026-08-08T05:59:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2365 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:48 dist-test-slave-zkpd krb5kdc[15368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:48 dist-test-slave-zkpd krb5kdc[15368](info): set up 2 sockets
Aug 08 05:59:48 dist-test-slave-zkpd krb5kdc[15368](info): commencing operation
krb5kdc: starting...
W20260808 05:59:49.169624 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T05:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T05:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 05:59:49.294230 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:44515
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:49.405611 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:49.405856 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:49.405927 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:49.409842 15381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 05:59:49.409889 15381 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 05:59:49.409906 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:49.409958 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:49.409991 15381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 05:59:49.410020 15381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 05:59:49.415189 15381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:44515
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:49.416189 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:49.417191 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:49.425393 15387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:49.425339 15386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:49.425475 15381 server_base.cc:1034] running on GCE node
W20260808 05:59:49.425336 15389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:49.426244 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:49.427296 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:49.428454 15381 hybrid_clock.cc:648] HybridClock initialized: now 1786168789428437 us; error 32 us; skew 500 ppm
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:49.435233 15381 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 05:59:49.436149 15381 webserver.cc:460] Webserver started at http://127.13.229.126:38323/ using document root <none> and password file <none>
I20260808 05:59:49.436694 15381 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:49.436755 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:49.437007 15381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:49.439004 15381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "35a872f1797f4a6abe460e057cfccab8"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "09508634c4455f8e547b4debdfe1b668"
server_key_iv: "2c57b887eed3c84858806ae72d18a9eb"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.439581 15381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "35a872f1797f4a6abe460e057cfccab8"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "09508634c4455f8e547b4debdfe1b668"
server_key_iv: "2c57b887eed3c84858806ae72d18a9eb"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.443193 15381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 05:59:49.445544 15396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:49.446791 15381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 05:59:49.446919 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "35a872f1797f4a6abe460e057cfccab8"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "09508634c4455f8e547b4debdfe1b668"
server_key_iv: "2c57b887eed3c84858806ae72d18a9eb"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.447014 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:49.460145 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:49.464474 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:49.464722 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:49.473126 15381 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:44515
I20260808 05:59:49.473145 15463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:44515 every 8 connection(s)
I20260808 05:59:49.474066 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 05:59:49.476838 15464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:49.480168 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15381
I20260808 05:59:49.480263 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 05:59:49.480526 14229 external_mini_cluster.cc:1428] Setting key 237aac1eee6f75a47e5167c1f5cb9c42
I20260808 05:59:49.482802 15464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Bootstrap starting.
I20260808 05:59:49.485495 15464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:49.486343 15464 log.cc:824] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:49.488560 15464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: No bootstrap required, opened a new log
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:49.492724 15464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } }
I20260808 05:59:49.493027 15464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:49.493098 15464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a872f1797f4a6abe460e057cfccab8, State: Initialized, Role: FOLLOWER
I20260808 05:59:49.493607 15464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } }
I20260808 05:59:49.493796 15464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 05:59:49.493891 15464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 05:59:49.493986 15464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:49.494985 15464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } }
I20260808 05:59:49.495352 15464 leader_election.cc:302] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a872f1797f4a6abe460e057cfccab8; no voters:
I20260808 05:59:49.495555 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.481928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.13.229.126:44515)
0808 05:59:49.482350 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:49.482369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:49.482411 (+ 42us) server_negotiation.cc:407] Connection header received
0808 05:59:49.483192 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:49.483217 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:49.483624 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:49.484006 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:49.484650 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.486149 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.487211 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.487868 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.488051 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.490394 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:49.490401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:49.490417 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:49.490448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:49.492409 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.492820 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.492826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.492831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.492967 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.493281 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.493285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.493286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.493789 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:49.494004 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:49.494341 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 05:59:49.494715 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260808 05:59:49.495695 15464 leader_election.cc:288] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 05:59:49.495832 15469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:49.496019 15469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [term 1 LEADER]: Becoming Leader. State: Replica: 35a872f1797f4a6abe460e057cfccab8, State: Running, Role: LEADER
I20260808 05:59:49.496379 15469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } }
I20260808 05:59:49.497006 15464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 05:59:49.497972 15471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a872f1797f4a6abe460e057cfccab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } } }
I20260808 05:59:49.498200 15471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:49.498477 15471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a872f1797f4a6abe460e057cfccab8. Latest consensus state: current_term: 1 leader_uuid: "35a872f1797f4a6abe460e057cfccab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a872f1797f4a6abe460e057cfccab8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44515 } } }
I20260808 05:59:49.498625 15471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8 [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:49.498641 15479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 05:59:49.502174 15479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 05:59:49.507843 15479 catalog_manager.cc:1347] Generated new cluster ID: 3bea8bed1f70405b8b2a669a99499a05
I20260808 05:59:49.507966 15479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 05:59:49.515717 15479 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 05:59:49.516901 15479 catalog_manager.cc:1504] Loading token signing keys...
I20260808 05:59:49.526811 15479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Generated new TSK 0
I20260808 05:59:49.527549 15479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 05:59:49.562232 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:44515
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:49.677712 15491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:49.677970 15491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:49.678016 15491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:49.682049 15491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:49.682106 15491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:49.682176 15491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:49.682279 15491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 05:59:49.687398 15491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:49.688508 15491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:49.689471 15491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:49.699355 15500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:49.699370 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:49.699362 15498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:49.699687 15491 server_base.cc:1034] running on GCE node
I20260808 05:59:49.700146 15491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:49.700805 15491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:49.701974 15491 hybrid_clock.cc:648] HybridClock initialized: now 1786168789701952 us; error 30 us; skew 500 ppm
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:49.704824 15491 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 05:59:49.705654 15491 webserver.cc:460] Webserver started at http://127.13.229.65:36549/ using document root <none> and password file <none>
I20260808 05:59:49.706214 15491 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:49.706306 15491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:49.706548 15491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:49.708636 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "d9ba0462fec34779878c3967a4a02e72"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "e3ca3e0d04784cd19227120bb334b4e7"
server_key_iv: "a4836ec8dcb149202f56a8d5faeb6d80"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.709199 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "d9ba0462fec34779878c3967a4a02e72"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "e3ca3e0d04784cd19227120bb334b4e7"
server_key_iv: "a4836ec8dcb149202f56a8d5faeb6d80"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.712980 15491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 05:59:49.715365 15507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:49.716566 15491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:49.716701 15491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "d9ba0462fec34779878c3967a4a02e72"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "e3ca3e0d04784cd19227120bb334b4e7"
server_key_iv: "a4836ec8dcb149202f56a8d5faeb6d80"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.716810 15491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:49.735519 15491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:49.738765 15491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:49.738965 15491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:49.740476 15491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:49.740554 15491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:49.740657 15491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:49.740703 15491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:49.751708 15491 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:44643
I20260808 05:59:49.752015 15631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:44643 every 8 connection(s)
I20260808 05:59:49.752905 15491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 05:59:49.755390 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.741257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:50941 (local address 127.13.229.126:44515)
0808 05:59:49.741389 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:49.741392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:49.742642 (+ 1250us) server_negotiation.cc:407] Connection header received
0808 05:59:49.743911 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:49.743915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:49.743975 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:49.744060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:49.745657 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.746580 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.748351 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.748733 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.748889 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.751694 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:49.751698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:49.751700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:49.751731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:49.753495 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.754118 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.754120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.754121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.754198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.754501 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.754503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.754504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.754672 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:49.754748 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:49.755173 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 05:59:49.755272 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 05:59:49.756240 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.742046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.65:50941)
0808 05:59:49.742484 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:49.742537 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:49.743649 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:49.744270 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:49.744279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:49.744726 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:49.745462 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.745482 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.746749 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.746753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.748220 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.748230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.748904 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.748908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.749049 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.749617 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:49.749643 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:49.751500 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:49.753684 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.753690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.753703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.754042 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.754290 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.754292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.754293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.754419 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.754760 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:49.754764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:49.754974 (+ 210us) client_negotiation.cc:769] Sending connection context
0808 05:59:49.755125 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 05:59:49.755366 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260808 05:59:49.757513 15632 heartbeater.cc:347] Connected to a master server at 127.13.229.126:44515
I20260808 05:59:49.757674 15632 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:49.758157 15632 heartbeater.cc:511] Master 127.13.229.126:44515 requested a full tablet report, sending...
I20260808 05:59:49.758772 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15491
I20260808 05:59:49.758899 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 05:59:49.759199 14229 external_mini_cluster.cc:1428] Setting key c9e014272e5266fbb80d3821991e9ecd
I20260808 05:59:49.759819 15413 ts_manager.cc:194] Registered new tserver with Master: d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643)
I20260808 05:59:49.761304 15413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:50941
I20260808 05:59:49.767154 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.760022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:46863 (local address 127.13.229.126:44515)
0808 05:59:49.760119 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:49.760123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:49.760138 (+ 15us) server_negotiation.cc:407] Connection header received
0808 05:59:49.760209 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:49.760214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:49.760278 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:49.760368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:49.760784 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.761378 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.762361 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.762576 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.762685 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.764022 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:49.764025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:49.764027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:49.764048 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:49.765481 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.765968 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.765972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.765974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.766056 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.766340 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.766344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.766346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.766523 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:49.766632 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:49.766888 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 05:59:49.766997 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 05:59:49.767179 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.759702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.65:46863)
0808 05:59:49.760026 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:49.760042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:49.760128 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:49.760386 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:49.760390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:49.760581 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:49.760667 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.760673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.761511 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.761514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.762243 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.762252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.762689 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.762692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.762847 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.763370 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:49.763391 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:49.763872 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:49.765645 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.765648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.765650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.765852 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.766152 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.766156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.766158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.766248 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.766623 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:49.766626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:49.766725 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 05:59:49.766832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 05:59:49.766984 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 05:59:49.789085 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:44515
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:49.903090 15642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:49.903445 15642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:49.903517 15642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:49.907610 15642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:49.907670 15642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:49.907708 15642 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:49.907745 15642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 05:59:49.913000 15642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:49.914029 15642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:49.915014 15642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:49.924882 15647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:49.924932 15648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:49.924950 15651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:49.925179 15642 server_base.cc:1034] running on GCE node
I20260808 05:59:49.925791 15642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:49.926429 15642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:49.927616 15642 hybrid_clock.cc:648] HybridClock initialized: now 1786168789927603 us; error 35 us; skew 500 ppm
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:49.930343 15642 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 05:59:49.931191 15642 webserver.cc:460] Webserver started at http://127.13.229.66:43393/ using document root <none> and password file <none>
I20260808 05:59:49.931742 15642 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:49.931834 15642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:49.932097 15642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:49.934110 15642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "a1679cd34a8b4af4a860cd6823d3cbb4"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "3f1afe41b5fbc156233b33f99ae753fa"
server_key_iv: "e24fbd63e760b01fba731783c967b4b3"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.934686 15642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "a1679cd34a8b4af4a860cd6823d3cbb4"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "3f1afe41b5fbc156233b33f99ae753fa"
server_key_iv: "e24fbd63e760b01fba731783c967b4b3"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.938369 15642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 05:59:49.940758 15659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:49.941964 15642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:49.942101 15642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "a1679cd34a8b4af4a860cd6823d3cbb4"
format_stamp: "Formatted at 2026-08-08 05:59:49 on dist-test-slave-zkpd"
server_key: "3f1afe41b5fbc156233b33f99ae753fa"
server_key_iv: "e24fbd63e760b01fba731783c967b4b3"
server_key_version: "encryptionkey@0"
I20260808 05:59:49.942209 15642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:49.957504 15642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:49.960620 15642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:49.960824 15642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:49.962257 15642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:49.962335 15642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:49.962407 15642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:49.962460 15642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:49 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:49.973623 15642 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35713
I20260808 05:59:49.973695 15784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35713 every 8 connection(s)
I20260808 05:59:49.974841 15642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 05:59:49.976174 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15642
I20260808 05:59:49.976300 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 05:59:49.976584 14229 external_mini_cluster.cc:1428] Setting key 1530d46b9fd1eb7c091119d3b0cd79d0
I20260808 05:59:49.977229 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.963177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:37969 (local address 127.13.229.126:44515)
0808 05:59:49.963356 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:49.963372 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:49.964192 (+ 820us) server_negotiation.cc:407] Connection header received
0808 05:59:49.965163 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:49.965167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:49.965227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:49.965317 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:49.966603 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.967331 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.968803 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.969031 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.969151 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.973144 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:49.973148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:49.973151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:49.973183 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:49.974926 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.975482 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.975485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.975487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.975581 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.976009 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.976015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.976019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.976286 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:49.976391 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:49.976946 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 05:59:49.977080 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 05:59:49.978144 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.963540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.66:37969)
0808 05:59:49.964030 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:49.964081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:49.964950 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:49.965492 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:49.965502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:49.965986 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:49.966455 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.966469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.967480 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.967483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.968644 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.968654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.969204 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.969208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.969392 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.970128 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:49.970154 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:49.972956 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:49.975049 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.975059 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.975070 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.975356 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.975671 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.975674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.975676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.975852 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.976399 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:49.976406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:49.976644 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 05:59:49.976843 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 05:59:49.977183 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260808 05:59:49.979431 15785 heartbeater.cc:347] Connected to a master server at 127.13.229.126:44515
I20260808 05:59:49.979600 15785 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:49.980221 15785 heartbeater.cc:511] Master 127.13.229.126:44515 requested a full tablet report, sending...
I20260808 05:59:49.981367 15413 ts_manager.cc:194] Registered new tserver with Master: a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713)
I20260808 05:59:49.982079 15413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:37969
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 05:59:49.989516 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.981916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.66:33729)
0808 05:59:49.982231 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:49.982246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:49.982359 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:49.982633 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:49.982636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:49.982855 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:49.982968 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.982976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.984128 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.984131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.984665 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:49.984673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.985330 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.985332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:49.985404 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.985930 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:49.985946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:49.986235 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:49.988186 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.988190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.988191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.988377 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.988616 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:49.988618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:49.988619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:49.988692 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:49.989117 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:49.989119 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:49.989196 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 05:59:49.989284 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 05:59:49.989390 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":116,"threads_started":1}
I20260808 05:59:49.989535 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:49.981972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:33729 (local address 127.13.229.126:44515)
0808 05:59:49.982073 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:49.982077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:49.982337 (+ 260us) server_negotiation.cc:407] Connection header received
0808 05:59:49.982459 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:49.982465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:49.982521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:49.982599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:49.983101 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.983994 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.984781 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:49.985202 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:49.985322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:49.986388 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:49.986393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:49.986395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:49.986422 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:49.988080 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.988459 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.988462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.988463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.988535 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:49.988767 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:49.988769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:49.988770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:49.989026 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:49.989107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:49.989338 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 05:59:49.989429 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 05:59:50.006374 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:44515
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:50.123777 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:50.124082 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:50.124130 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:50.127976 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:50.128023 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:50.128060 15795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:50.128093 15795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 05:59:50.133574 15795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:50.134589 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:50.135562 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:50.145130 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:50.145153 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:50.145113 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:50.145196 15795 server_base.cc:1034] running on GCE node
I20260808 05:59:50.145730 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:50.146436 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:50.147593 15795 hybrid_clock.cc:648] HybridClock initialized: now 1786168790147563 us; error 42 us; skew 500 ppm
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168790, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:50.150537 15795 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 05:59:50.151420 15795 webserver.cc:460] Webserver started at http://127.13.229.67:33521/ using document root <none> and password file <none>
I20260808 05:59:50.152007 15795 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:50.152073 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:50.152354 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:50.154421 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "c65ca44e814e4dfa85c0e108ef27af99"
format_stamp: "Formatted at 2026-08-08 05:59:50 on dist-test-slave-zkpd"
server_key: "9bf02a412bc39e857cfd767da569faa8"
server_key_iv: "82cbc5aea839d0ee7b52d8a967c1e9a0"
server_key_version: "encryptionkey@0"
I20260808 05:59:50.154994 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "c65ca44e814e4dfa85c0e108ef27af99"
format_stamp: "Formatted at 2026-08-08 05:59:50 on dist-test-slave-zkpd"
server_key: "9bf02a412bc39e857cfd767da569faa8"
server_key_iv: "82cbc5aea839d0ee7b52d8a967c1e9a0"
server_key_version: "encryptionkey@0"
I20260808 05:59:50.158860 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 05:59:50.161434 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:50.162823 15795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 05:59:50.162958 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "c65ca44e814e4dfa85c0e108ef27af99"
format_stamp: "Formatted at 2026-08-08 05:59:50 on dist-test-slave-zkpd"
server_key: "9bf02a412bc39e857cfd767da569faa8"
server_key_iv: "82cbc5aea839d0ee7b52d8a967c1e9a0"
server_key_version: "encryptionkey@0"
I20260808 05:59:50.163066 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:50.178969 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:50.182639 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:50.182967 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:50.185042 15795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:50.185110 15795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:50.185195 15795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:50.185237 15795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168790, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:50.196338 15795 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:33427
I20260808 05:59:50.196434 15933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:33427 every 8 connection(s)
I20260808 05:59:50.197402 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 05:59:50.199787 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.186142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:38761 (local address 127.13.229.126:44515)
0808 05:59:50.186275 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.186279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.187358 (+ 1079us) server_negotiation.cc:407] Connection header received
0808 05:59:50.188581 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.188585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.188647 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.188776 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.190440 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.191360 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.192737 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.193043 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.193162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.195962 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.195969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.195974 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.196001 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.197625 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.198274 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.198278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.198280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.198371 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.198763 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.198766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.198767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.198940 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.199027 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.199546 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.199657 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 05:59:50.200518 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.186692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.67:38761)
0808 05:59:50.187188 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.187248 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.188329 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.188997 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.189011 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.189498 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:50.190250 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.190277 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.191505 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.191508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.192599 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.192610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.193207 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.193211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.193403 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.193987 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:50.194003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:50.195768 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:50.197780 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.197785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.197800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.198175 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.198470 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.198473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.198474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.198668 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.199039 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:50.199044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:50.199272 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.199429 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.199667 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":139,"threads_started":1}
I20260808 05:59:50.201737 15934 heartbeater.cc:347] Connected to a master server at 127.13.229.126:44515
I20260808 05:59:50.201910 15934 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:50.202476 15934 heartbeater.cc:511] Master 127.13.229.126:44515 requested a full tablet report, sending...
I20260808 05:59:50.203302 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 15795
I20260808 05:59:50.203441 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 05:59:50.203593 15414 ts_manager.cc:194] Registered new tserver with Master: c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427)
I20260808 05:59:50.203740 14229 external_mini_cluster.cc:1428] Setting key b1da006b01e9b4af56d75c578f43d082
I20260808 05:59:50.204454 15414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:38761
I20260808 05:59:50.205824 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 05:59:50.213274 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.204118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.13.229.67:46565)
0808 05:59:50.204915 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.204931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.205602 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.205889 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.205893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.206110 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:50.206240 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.206247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.207036 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.207044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.208174 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.208191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.208830 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.208833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.208918 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.209461 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:50.209474 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:50.209842 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:50.211610 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.211617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.211622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.211942 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.212233 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.212236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.212239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.212347 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.212829 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:50.212832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:50.212924 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.213010 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.213121 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":108,"threads_started":1}
I20260808 05:59:50.213479 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.204022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:46565 (local address 127.13.229.126:44515)
0808 05:59:50.204155 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.204160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.205002 (+ 842us) server_negotiation.cc:407] Connection header received
0808 05:59:50.205713 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.205716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.205762 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.205836 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.206381 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.206893 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.208311 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.208697 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.208811 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.210011 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.210017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.210024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.210045 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.211495 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.212052 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.212055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.212056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.212128 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.212477 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.212480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.212482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.212718 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.212830 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.213122 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.213260 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:50.235421 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.227032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.13.229.126:44515)
0808 05:59:50.227157 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.227160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.227173 (+ 13us) server_negotiation.cc:407] Connection header received
0808 05:59:50.227213 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.227215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.227254 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.227330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.228171 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.228719 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.229566 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.229783 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.229868 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.232393 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.232397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.232399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.232424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.233927 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.234288 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.234290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.234291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.234365 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.234592 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.234594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.234595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.234776 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.234883 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.235089 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.235183 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 05:59:50.238584 15414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 05:59:50.241775 15414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:50.258033 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.250869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33427 (local address 127.0.0.1:36576)
0808 05:59:50.251941 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.251958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.252050 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.252803 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.252809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.252831 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.252989 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.252996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.254514 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.254517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.256223 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.256235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.257235 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.257239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.257350 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.257405 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.257538 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.257677 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":328,"thread_start_us":60,"threads_started":1}
I20260808 05:59:50.258245 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.251144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35713 (local address 127.0.0.1:46214)
0808 05:59:50.251823 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.251876 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.252038 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.252973 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.252976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.253005 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.253243 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.253250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.255716 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.255721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.256714 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.256726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.257726 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.257739 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.257841 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.257873 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.257949 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.258016 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":82,"spinlock_wait_cycles":5248,"thread_start_us":130,"threads_started":1}
I20260808 05:59:50.258811 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.251379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.13.229.67:33427)
0808 05:59:50.252362 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.252375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.252407 (+ 32us) server_negotiation.cc:407] Connection header received
0808 05:59:50.252482 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.252486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.252638 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.252814 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.253105 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.254393 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.256356 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.257118 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.257206 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.257344 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.258553 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.258656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":122,"threads_started":1}
I20260808 05:59:50.259044 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.251635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.13.229.66:35713)
0808 05:59:50.252317 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.252333 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.252359 (+ 26us) server_negotiation.cc:407] Connection header received
0808 05:59:50.252440 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.252448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.252656 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.252807 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.254690 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.255585 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.256889 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.257608 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.257710 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.257873 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.258796 (+ 923us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.258907 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":97,"threads_started":1}
I20260808 05:59:50.260283 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.251264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:44643 (local address 127.0.0.1:60004)
0808 05:59:50.252293 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.252306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.252415 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.252975 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.252977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.253007 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.253243 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.253250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.254650 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.254653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.256223 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.256235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.259854 (+ 3619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.259858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.259978 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.260000 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.260091 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.260156 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":101,"spinlock_wait_cycles":4992,"thread_start_us":111,"threads_started":1}
I20260808 05:59:50.260681 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.251716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.13.229.65:44643)
0808 05:59:50.252163 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.252179 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.252506 (+ 327us) server_negotiation.cc:407] Connection header received
0808 05:59:50.252578 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.252583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.252733 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.252853 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.253374 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.254545 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.257460 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.259721 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.259855 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.260014 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.260488 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.260584 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":63,"threads_started":1}
I20260808 05:59:50.261873 15863 tablet_service.cc:1467] Processing CreateTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 (DEFAULT_TABLE table=test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.261863 15715 tablet_service.cc:1467] Processing CreateTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 (DEFAULT_TABLE table=test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.263016 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e5bbf34c340ecb05dce6e924f3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.263020 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e5bbf34c340ecb05dce6e924f3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.263149 15562 tablet_service.cc:1467] Processing CreateTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 (DEFAULT_TABLE table=test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.264000 15562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e5bbf34c340ecb05dce6e924f3a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.271049 15958 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Bootstrap starting.
I20260808 05:59:50.272195 15959 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Bootstrap starting.
I20260808 05:59:50.273368 15958 tablet_bootstrap.cc:654] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.273940 15959 tablet_bootstrap.cc:654] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.274487 15958 log.cc:824] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:50.274657 15959 log.cc:824] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:50.275377 15960 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Bootstrap starting.
I20260808 05:59:50.276507 15958 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: No bootstrap required, opened a new log
I20260808 05:59:50.276713 15958 ts_tablet_manager.cc:1397] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260808 05:59:50.277221 15959 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: No bootstrap required, opened a new log
I20260808 05:59:50.277395 15960 tablet_bootstrap.cc:654] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.277431 15959 ts_tablet_manager.cc:1397] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 05:59:50.278200 15960 log.cc:824] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:50.280126 15958 raft_consensus.cc:348] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.280339 15960 tablet_bootstrap.cc:492] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: No bootstrap required, opened a new log
I20260808 05:59:50.280381 15958 raft_consensus.cc:374] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.280449 15958 raft_consensus.cc:729] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ba0462fec34779878c3967a4a02e72, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.280511 15960 ts_tablet_manager.cc:1397] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 05:59:50.280925 15958 consensus_queue.cc:260] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.281159 15959 raft_consensus.cc:348] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.281428 15959 raft_consensus.cc:374] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.281491 15959 raft_consensus.cc:729] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ca44e814e4dfa85c0e108ef27af99, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.281631 15632 heartbeater.cc:503] Master 127.13.229.126:44515 was elected leader, sending a full tablet report...
I20260808 05:59:50.281912 15958 ts_tablet_manager.cc:1428] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260808 05:59:50.282616 15959 consensus_queue.cc:260] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.283689 15959 ts_tablet_manager.cc:1428] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260808 05:59:50.284058 15934 heartbeater.cc:503] Master 127.13.229.126:44515 was elected leader, sending a full tablet report...
I20260808 05:59:50.284086 15960 raft_consensus.cc:348] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.284346 15960 raft_consensus.cc:374] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.284407 15960 raft_consensus.cc:729] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1679cd34a8b4af4a860cd6823d3cbb4, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.284825 15960 consensus_queue.cc:260] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.285583 15785 heartbeater.cc:503] Master 127.13.229.126:44515 was elected leader, sending a full tablet report...
I20260808 05:59:50.285815 15960 ts_tablet_manager.cc:1428] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 05:59:50.325085 15966 raft_consensus.cc:482] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:50.325314 15966 raft_consensus.cc:504] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.326692 15966 leader_election.cc:288] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427), d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643)
I20260808 05:59:50.331318 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.326905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33427 (local address 127.13.229.66:60195)
0808 05:59:50.327051 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.327066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.327164 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.327487 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.327490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.327528 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.327727 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.327734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.328711 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.328718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.330040 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.330051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.330937 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.330939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.331034 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.331055 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.331115 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.331172 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1,"spinlock_wait_cycles":21248}
I20260808 05:59:50.331422 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.327110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:60195 (local address 127.13.229.67:33427)
0808 05:59:50.327247 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.327254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.327270 (+ 16us) server_negotiation.cc:407] Connection header received
0808 05:59:50.327326 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.327329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.327375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.327459 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.327909 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.328598 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.330189 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.330809 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.330947 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.331006 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.331159 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.331262 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 05:59:50.331799 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.327099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:41679 (local address 127.13.229.65:44643)
0808 05:59:50.327238 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.327243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.327547 (+ 304us) server_negotiation.cc:407] Connection header received
0808 05:59:50.327672 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.327675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.327726 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.327822 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.328144 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.328723 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.330492 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.331192 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.331320 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.331401 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.331593 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.331668 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 05:59:50.332197 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.327021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:44643 (local address 127.13.229.66:41679)
0808 05:59:50.327453 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.327466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.327562 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.327860 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.327863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.327881 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.328041 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.328048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.328831 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.328834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.330359 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.330368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.331324 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.331331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.331430 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.331459 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.331997 (+ 538us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.332076 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"thread_start_us":158,"threads_started":1}
I20260808 05:59:50.332146 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e5bbf34c340ecb05dce6e924f3a01" candidate_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ca44e814e4dfa85c0e108ef27af99" is_pre_election: true
I20260808 05:59:50.332484 15885 raft_consensus.cc:2454] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1679cd34a8b4af4a860cd6823d3cbb4 in term 0.
I20260808 05:59:50.332907 15583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e5bbf34c340ecb05dce6e924f3a01" candidate_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ba0462fec34779878c3967a4a02e72" is_pre_election: true
I20260808 05:59:50.333032 15664 leader_election.cc:302] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1679cd34a8b4af4a860cd6823d3cbb4, c65ca44e814e4dfa85c0e108ef27af99; no voters:
I20260808 05:59:50.333202 15583 raft_consensus.cc:2454] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1679cd34a8b4af4a860cd6823d3cbb4 in term 0.
I20260808 05:59:50.333339 15966 raft_consensus.cc:2781] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:50.333465 15966 raft_consensus.cc:482] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:50.333536 15966 raft_consensus.cc:3037] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.334751 15966 raft_consensus.cc:504] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.335099 15966 leader_election.cc:288] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [CANDIDATE]: Term 1 election: Requested vote from peers c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427), d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643)
I20260808 05:59:50.335412 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e5bbf34c340ecb05dce6e924f3a01" candidate_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ca44e814e4dfa85c0e108ef27af99"
I20260808 05:59:50.335460 15583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e5bbf34c340ecb05dce6e924f3a01" candidate_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ba0462fec34779878c3967a4a02e72"
I20260808 05:59:50.335536 15885 raft_consensus.cc:3037] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.335597 15583 raft_consensus.cc:3037] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.336701 15583 raft_consensus.cc:2454] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1679cd34a8b4af4a860cd6823d3cbb4 in term 1.
I20260808 05:59:50.336766 15885 raft_consensus.cc:2454] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1679cd34a8b4af4a860cd6823d3cbb4 in term 1.
I20260808 05:59:50.337064 15663 leader_election.cc:302] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1679cd34a8b4af4a860cd6823d3cbb4, d9ba0462fec34779878c3967a4a02e72; no voters:
I20260808 05:59:50.337227 15966 raft_consensus.cc:2781] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:50.337497 15966 raft_consensus.cc:686] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 LEADER]: Becoming Leader. State: Replica: a1679cd34a8b4af4a860cd6823d3cbb4, State: Running, Role: LEADER
I20260808 05:59:50.337817 15966 consensus_queue.cc:237] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } }
I20260808 05:59:50.341151 15413 catalog_manager.cc:5495] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66). New cstate: current_term: 1 leader_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } health_report { overall_health: UNKNOWN } } }
I20260808 05:59:50.361680 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.357956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.13.229.66:35713)
0808 05:59:50.358088 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.358091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.358151 (+ 60us) server_negotiation.cc:407] Connection header received
0808 05:59:50.358337 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.358340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.358397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.358484 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:50.358729 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.359292 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.360289 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.360504 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.360586 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.360926 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.361317 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.361404 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.361500 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.361551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 05:59:50.367349 15583 raft_consensus.cc:1261] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Refusing update from remote peer a1679cd34a8b4af4a860cd6823d3cbb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.367681 15885 raft_consensus.cc:1261] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Refusing update from remote peer a1679cd34a8b4af4a860cd6823d3cbb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.368213 15968 consensus_queue.cc:1035] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.368402 15966 consensus_queue.cc:1035] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.375526 15971 mvcc.cc:205] Tried to move back new op lower bound from 7316147365337636864 to 7316147365226586112. Current Snapshot: MvccSnapshot[applied={T|T < 7316147365337636864}]
I20260808 05:59:50.376652 15973 mvcc.cc:205] Tried to move back new op lower bound from 7316147365337636864 to 7316147365226586112. Current Snapshot: MvccSnapshot[applied={T|T < 7316147365337636864}]
I20260808 05:59:50.380947 15972 mvcc.cc:205] Tried to move back new op lower bound from 7316147365337636864 to 7316147365226586112. Current Snapshot: MvccSnapshot[applied={T|T < 7316147365337636864}]
I20260808 05:59:50.387557 15414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:50.387730 15414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:50.390115 15414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Sending DeleteTablet for 3 replicas of tablet e62e5bbf34c340ecb05dce6e924f3a01
I20260808 05:59:50.390937 15715 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:46214
I20260808 05:59:50.390898 15863 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36576
I20260808 05:59:50.391088 15562 tablet_service.cc:1514] Processing DeleteTablet for tablet e62e5bbf34c340ecb05dce6e924f3a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:60004
I20260808 05:59:50.391588 15985 tablet_replica.cc:330] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: stopping tablet replica
I20260808 05:59:50.391732 15984 tablet_replica.cc:330] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: stopping tablet replica
I20260808 05:59:50.391994 15985 raft_consensus.cc:2229] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:50.392036 15984 raft_consensus.cc:2229] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:50.392309 15985 raft_consensus.cc:2258] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.392344 15984 raft_consensus.cc:2258] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.392416 15414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 05:59:50.392740 15983 tablet_replica.cc:330] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: stopping tablet replica
W20260808 05:59:50.392815 15414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:50.393003 15983 raft_consensus.cc:2229] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 05:59:50.393338 15983 raft_consensus.cc:2258] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.394153 15984 ts_tablet_manager.cc:1905] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.395393 15983 ts_tablet_manager.cc:1905] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.396958 15984 ts_tablet_manager.cc:1918] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 05:59:50.397053 15984 log.cc:1196] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/e62e5bbf34c340ecb05dce6e924f3a01
I20260808 05:59:50.398252 15983 ts_tablet_manager.cc:1918] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 05:59:50.398348 15983 log.cc:1196] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/e62e5bbf34c340ecb05dce6e924f3a01
I20260808 05:59:50.398509 15984 ts_tablet_manager.cc:1939] T e62e5bbf34c340ecb05dce6e924f3a01 P c65ca44e814e4dfa85c0e108ef27af99: Deleting consensus metadata
I20260808 05:59:50.398681 15983 ts_tablet_manager.cc:1939] T e62e5bbf34c340ecb05dce6e924f3a01 P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting consensus metadata
I20260808 05:59:50.399206 15985 ts_tablet_manager.cc:1905] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.399771 15562 tablet_service.cc:1467] Processing CreateTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd (DEFAULT_TABLE table=test-table [id=c668ad689a81442dacea8cd4d9a420f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.399771 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd (DEFAULT_TABLE table=test-table [id=c668ad689a81442dacea8cd4d9a420f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.400036 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522eb3ebe37848638c3e6c80a9cdf1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.400038 15562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522eb3ebe37848638c3e6c80a9cdf1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.402204 15958 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Bootstrap starting.
I20260808 05:59:50.402295 15960 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Bootstrap starting.
I20260808 05:59:50.402581 15985 ts_tablet_manager.cc:1918] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 05:59:50.402570 15400 catalog_manager.cc:4841] TS c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427): tablet e62e5bbf34c340ecb05dce6e924f3a01 (table test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]) successfully deleted
I20260808 05:59:50.402667 15985 log.cc:1196] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/e62e5bbf34c340ecb05dce6e924f3a01
I20260808 05:59:50.403098 15985 ts_tablet_manager.cc:1939] T e62e5bbf34c340ecb05dce6e924f3a01 P d9ba0462fec34779878c3967a4a02e72: Deleting consensus metadata
I20260808 05:59:50.403151 15863 tablet_service.cc:1467] Processing CreateTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd (DEFAULT_TABLE table=test-table [id=c668ad689a81442dacea8cd4d9a420f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:50.403304 15960 tablet_bootstrap.cc:654] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.403333 15958 tablet_bootstrap.cc:654] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.403409 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522eb3ebe37848638c3e6c80a9cdf1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.404350 15397 catalog_manager.cc:4841] TS a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713): tablet e62e5bbf34c340ecb05dce6e924f3a01 (table test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]) successfully deleted
I20260808 05:59:50.404785 15399 catalog_manager.cc:4841] TS d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643): tablet e62e5bbf34c340ecb05dce6e924f3a01 (table test-table [id=17dcc9993ba24c7eb08c6b9225cff34f]) successfully deleted
I20260808 05:59:50.405232 15960 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: No bootstrap required, opened a new log
I20260808 05:59:50.405314 15960 ts_tablet_manager.cc:1397] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 05:59:50.405566 15958 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: No bootstrap required, opened a new log
I20260808 05:59:50.405637 15958 ts_tablet_manager.cc:1397] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 05:59:50.405848 15959 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Bootstrap starting.
I20260808 05:59:50.405814 15960 raft_consensus.cc:348] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.405939 15960 raft_consensus.cc:374] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.405987 15960 raft_consensus.cc:729] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1679cd34a8b4af4a860cd6823d3cbb4, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.406097 15960 consensus_queue.cc:260] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.406334 15960 ts_tablet_manager.cc:1428] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:50.406366 15958 raft_consensus.cc:348] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.406487 15958 raft_consensus.cc:374] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.406519 15958 raft_consensus.cc:729] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ba0462fec34779878c3967a4a02e72, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.406627 15958 consensus_queue.cc:260] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.406857 15959 tablet_bootstrap.cc:654] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.407064 15958 ts_tablet_manager.cc:1428] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260808 05:59:50.408195 15959 tablet_bootstrap.cc:492] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: No bootstrap required, opened a new log
I20260808 05:59:50.408308 15959 ts_tablet_manager.cc:1397] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 05:59:50.408779 15959 raft_consensus.cc:348] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.408896 15959 raft_consensus.cc:374] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.408946 15959 raft_consensus.cc:729] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ca44e814e4dfa85c0e108ef27af99, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.409073 15959 consensus_queue.cc:260] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.409462 15959 ts_tablet_manager.cc:1428] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260808 05:59:50.448465 15935 tablet.cc:2369] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 05:59:50.476167 15787 tablet.cc:2369] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 05:59:50.504225 15633 tablet.cc:2369] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 05:59:50.609850 15965 raft_consensus.cc:482] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:50.610045 15965 raft_consensus.cc:504] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.611192 15965 leader_election.cc:288] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643), a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713)
I20260808 05:59:50.614768 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.611556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54401 (local address 127.13.229.65:44643)
0808 05:59:50.611682 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.611687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.611701 (+ 14us) server_negotiation.cc:407] Connection header received
0808 05:59:50.611845 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.611848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.611948 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.612026 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.612296 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.612793 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.613830 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.614210 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.614337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.614394 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.614574 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.614651 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 05:59:50.614768 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.611411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:44643 (local address 127.13.229.67:54401)
0808 05:59:50.611608 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.611624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.611722 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.612051 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.612054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.612072 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.612193 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.612198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.612914 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.612917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.613695 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.613704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.614327 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.614330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.614453 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.614471 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.614557 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.614627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 05:59:50.615033 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.611442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35713 (local address 127.13.229.67:34495)
0808 05:59:50.611816 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.611827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.612000 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.612270 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.612279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.612295 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.612423 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.612428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.613127 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.613132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.613948 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.613957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.614605 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.614609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.614717 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.614743 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.614832 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.614895 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":91,"threads_started":1}
I20260808 05:59:50.615063 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.611570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:34495 (local address 127.13.229.66:35713)
0808 05:59:50.611703 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.611707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.611943 (+ 236us) server_negotiation.cc:407] Connection header received
0808 05:59:50.612122 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.612125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.612171 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.612247 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.612535 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.613017 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.614077 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.614466 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.614592 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.614652 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.614853 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.614920 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 05:59:50.615250 15583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522eb3ebe37848638c3e6c80a9cdf1bd" candidate_uuid: "c65ca44e814e4dfa85c0e108ef27af99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ba0462fec34779878c3967a4a02e72" is_pre_election: true
I20260808 05:59:50.615430 15583 raft_consensus.cc:2454] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c65ca44e814e4dfa85c0e108ef27af99 in term 0.
I20260808 05:59:50.615553 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522eb3ebe37848638c3e6c80a9cdf1bd" candidate_uuid: "c65ca44e814e4dfa85c0e108ef27af99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" is_pre_election: true
I20260808 05:59:50.615816 15737 raft_consensus.cc:2454] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c65ca44e814e4dfa85c0e108ef27af99 in term 0.
I20260808 05:59:50.615814 15813 leader_election.cc:302] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c65ca44e814e4dfa85c0e108ef27af99, d9ba0462fec34779878c3967a4a02e72; no voters:
I20260808 05:59:50.616106 15965 raft_consensus.cc:2781] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:50.616180 15965 raft_consensus.cc:482] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:50.616235 15965 raft_consensus.cc:3037] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.617257 15965 raft_consensus.cc:504] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.617631 15965 leader_election.cc:288] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [CANDIDATE]: Term 1 election: Requested vote from peers d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643), a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713)
I20260808 05:59:50.618006 15583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522eb3ebe37848638c3e6c80a9cdf1bd" candidate_uuid: "c65ca44e814e4dfa85c0e108ef27af99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9ba0462fec34779878c3967a4a02e72"
I20260808 05:59:50.618039 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522eb3ebe37848638c3e6c80a9cdf1bd" candidate_uuid: "c65ca44e814e4dfa85c0e108ef27af99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4"
I20260808 05:59:50.618131 15583 raft_consensus.cc:3037] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.618158 15737 raft_consensus.cc:3037] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.619042 15583 raft_consensus.cc:2454] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65ca44e814e4dfa85c0e108ef27af99 in term 1.
I20260808 05:59:50.619192 15737 raft_consensus.cc:2454] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c65ca44e814e4dfa85c0e108ef27af99 in term 1.
I20260808 05:59:50.619326 15813 leader_election.cc:302] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c65ca44e814e4dfa85c0e108ef27af99, d9ba0462fec34779878c3967a4a02e72; no voters:
I20260808 05:59:50.619513 15965 raft_consensus.cc:2781] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:50.619779 15965 raft_consensus.cc:686] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 1 LEADER]: Becoming Leader. State: Replica: c65ca44e814e4dfa85c0e108ef27af99, State: Running, Role: LEADER
I20260808 05:59:50.620047 15965 consensus_queue.cc:237] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.622344 15414 catalog_manager.cc:5495] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 reported cstate change: term changed from 0 to 1, leader changed from <none> to c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "c65ca44e814e4dfa85c0e108ef27af99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } health_report { overall_health: HEALTHY } } }
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:50.677767 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.669913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.13.229.126:44515)
0808 05:59:50.670071 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.670075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.670182 (+ 107us) server_negotiation.cc:407] Connection header received
0808 05:59:50.670294 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.670297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.670338 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.670417 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.671069 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.671571 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.672420 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.672630 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.672702 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.674964 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.674967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.674969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.674990 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.676240 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.676680 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.676682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.676684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.676755 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.677018 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.677020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.677021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.677176 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.677251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.677523 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.677642 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":75}
I20260808 05:59:50.677750 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.669821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44515 (local address 127.0.0.1:49294)
0808 05:59:50.670099 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.670113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.670216 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.670449 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.670451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.670875 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:50.670971 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.670977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.671660 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.671662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.672313 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.672322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.672716 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.672737 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.672821 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.673418 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:50.673430 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:50.674840 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:50.676353 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.676361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.676366 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.676595 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.676849 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:50.676851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:50.676853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:50.676939 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:50.677262 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:50.677268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:50.677377 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.677469 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.677588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260808 05:59:50.681635 15414 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:49294:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 05:59:50.682396 15414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:50.687194 15715 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0451dabc4d4256a307c3826511cbb5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce5480db95ea4f049cbc9fbba7c52ab6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:50.687209 15863 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0451dabc4d4256a307c3826511cbb5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce5480db95ea4f049cbc9fbba7c52ab6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:50.687217 15562 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0451dabc4d4256a307c3826511cbb5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce5480db95ea4f049cbc9fbba7c52ab6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:50.687625 15562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0451dabc4d4256a307c3826511cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.687664 15715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0451dabc4d4256a307c3826511cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.687695 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0451dabc4d4256a307c3826511cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:50.690353 15958 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: Bootstrap starting.
I20260808 05:59:50.690745 15960 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: Bootstrap starting.
I20260808 05:59:50.690789 15959 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: Bootstrap starting.
I20260808 05:59:50.691382 15958 tablet_bootstrap.cc:654] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.691552 15959 tablet_bootstrap.cc:654] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.691651 15960 tablet_bootstrap.cc:654] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:50.692888 15960 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: No bootstrap required, opened a new log
I20260808 05:59:50.693005 15960 ts_tablet_manager.cc:1397] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 05:59:50.693038 15958 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: No bootstrap required, opened a new log
I20260808 05:59:50.693116 15958 ts_tablet_manager.cc:1397] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 05:59:50.693450 15960 raft_consensus.cc:348] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.693503 15958 raft_consensus.cc:348] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.693578 15960 raft_consensus.cc:374] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.693586 15958 raft_consensus.cc:374] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.693634 15958 raft_consensus.cc:729] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ba0462fec34779878c3967a4a02e72, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.693630 15960 raft_consensus.cc:729] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1679cd34a8b4af4a860cd6823d3cbb4, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.693775 15958 consensus_queue.cc:260] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.693790 15960 consensus_queue.cc:260] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.694073 15958 ts_tablet_manager.cc:1428] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 05:59:50.694031 15964 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.694103 15960 ts_tablet_manager.cc:1428] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 05:59:50.694077 15966 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.694154 15964 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.694203 15966 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.695134 15959 tablet_bootstrap.cc:492] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: No bootstrap required, opened a new log
I20260808 05:59:50.695230 15959 ts_tablet_manager.cc:1397] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 05:59:50.695785 15959 raft_consensus.cc:348] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.695943 15959 raft_consensus.cc:374] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:50.695987 15959 raft_consensus.cc:729] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65ca44e814e4dfa85c0e108ef27af99, State: Initialized, Role: FOLLOWER
I20260808 05:59:50.696194 15959 consensus_queue.cc:260] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.696513 15959 ts_tablet_manager.cc:1428] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:50.696461 15965 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.696635 15965 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.792428 15964 raft_consensus.cc:482] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:50.792577 15964 raft_consensus.cc:504] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.793756 15964 leader_election.cc:288] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713), c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427)
I20260808 05:59:50.799216 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.794073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:45023 (local address 127.13.229.66:35713)
0808 05:59:50.794206 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.794211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.794450 (+ 239us) server_negotiation.cc:407] Connection header received
0808 05:59:50.794604 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.794607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.794660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.794747 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.795120 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.795963 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.797863 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.798493 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.798627 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.798681 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.799016 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.799091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 05:59:50.799216 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.794137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:39601 (local address 127.13.229.67:33427)
0808 05:59:50.794268 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.794273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.794444 (+ 171us) server_negotiation.cc:407] Connection header received
0808 05:59:50.794604 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.794607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.794660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.794747 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:50.795120 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.795940 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.797840 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.798493 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.798630 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.798681 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.799000 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.799083 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 05:59:50.799268 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.793950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35713 (local address 127.13.229.65:45023)
0808 05:59:50.794330 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.794345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.794488 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.794768 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.794771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.794796 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.795003 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.795010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.796067 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.796071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.797684 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.797692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.798613 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.798616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.798845 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.798883 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.798991 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.799082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":271,"spinlock_wait_cycles":18944,"thread_start_us":135,"threads_started":1}
I20260808 05:59:50.799268 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.794042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33427 (local address 127.13.229.65:39601)
0808 05:59:50.794330 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.794345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.794488 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.794768 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.794771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.794796 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:50.795003 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.795010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.796076 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.796079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.797680 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.797692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.798622 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.798624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.798841 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.798894 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.798975 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.799079 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":276,"spinlock_wait_cycles":25344,"thread_start_us":100,"threads_started":1}
I20260808 05:59:50.799769 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0451dabc4d4256a307c3826511cbb5" candidate_uuid: "d9ba0462fec34779878c3967a4a02e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" is_pre_election: true
I20260808 05:59:50.799991 15737 raft_consensus.cc:2454] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ba0462fec34779878c3967a4a02e72 in term 0.
I20260808 05:59:50.800246 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0451dabc4d4256a307c3826511cbb5" candidate_uuid: "d9ba0462fec34779878c3967a4a02e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ca44e814e4dfa85c0e108ef27af99" is_pre_election: true
I20260808 05:59:50.800395 15885 raft_consensus.cc:2454] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ba0462fec34779878c3967a4a02e72 in term 0.
I20260808 05:59:50.800791 15511 leader_election.cc:302] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c65ca44e814e4dfa85c0e108ef27af99, d9ba0462fec34779878c3967a4a02e72; no voters:
I20260808 05:59:50.801071 15964 raft_consensus.cc:2781] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:50.801142 15964 raft_consensus.cc:482] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:50.801196 15964 raft_consensus.cc:3037] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.802659 15964 raft_consensus.cc:504] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.803053 15964 leader_election.cc:288] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [CANDIDATE]: Term 1 election: Requested vote from peers a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713), c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427)
I20260808 05:59:50.803416 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0451dabc4d4256a307c3826511cbb5" candidate_uuid: "d9ba0462fec34779878c3967a4a02e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65ca44e814e4dfa85c0e108ef27af99"
I20260808 05:59:50.803416 15737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0451dabc4d4256a307c3826511cbb5" candidate_uuid: "d9ba0462fec34779878c3967a4a02e72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4"
I20260808 05:59:50.803540 15885 raft_consensus.cc:3037] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.803540 15737 raft_consensus.cc:3037] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:50.804733 15737 raft_consensus.cc:2454] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ba0462fec34779878c3967a4a02e72 in term 1.
I20260808 05:59:50.804733 15885 raft_consensus.cc:2454] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ba0462fec34779878c3967a4a02e72 in term 1.
I20260808 05:59:50.805039 15508 leader_election.cc:302] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1679cd34a8b4af4a860cd6823d3cbb4, d9ba0462fec34779878c3967a4a02e72; no voters:
I20260808 05:59:50.805266 15964 raft_consensus.cc:2781] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:50.805490 15964 raft_consensus.cc:686] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [term 1 LEADER]: Becoming Leader. State: Replica: d9ba0462fec34779878c3967a4a02e72, State: Running, Role: LEADER
I20260808 05:59:50.805734 15964 consensus_queue.cc:237] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } }
I20260808 05:59:50.806869 16005 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: TxnStatusTablet state changed. Reason: New leader d9ba0462fec34779878c3967a4a02e72. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.807005 16005 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:50.807269 16007 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 05:59:50.807433 16007 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 05:59:50.808245 15412 catalog_manager.cc:5495] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9ba0462fec34779878c3967a4a02e72 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } health_report { overall_health: UNKNOWN } } }
I20260808 05:59:50.847363 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.843239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.13.229.65:44643)
0808 05:59:50.843359 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.843371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.843447 (+ 76us) server_negotiation.cc:407] Connection header received
0808 05:59:50.843660 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.843662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.843721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.843808 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:50.844098 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.844632 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.845899 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.846116 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.846200 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.846550 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.846989 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.847080 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.847200 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.847251 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 05:59:50.847417 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.843122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:44643 (local address 127.0.0.1:33446)
0808 05:59:50.843371 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.843387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.843507 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.843848 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.843851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.843860 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:50.843994 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.844001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.844754 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.844757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.845756 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.845787 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.846227 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.846231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.846323 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.846378 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.847088 (+ 710us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.847103 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.847180 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.847274 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260808 05:59:50.850780 15885 raft_consensus.cc:1261] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Refusing update from remote peer d9ba0462fec34779878c3967a4a02e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.850883 15737 raft_consensus.cc:1261] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Refusing update from remote peer d9ba0462fec34779878c3967a4a02e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.851272 16005 consensus_queue.cc:1035] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.851435 15964 consensus_queue.cc:1035] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.852820 15965 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: TxnStatusTablet state changed. Reason: New leader d9ba0462fec34779878c3967a4a02e72. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.852929 15965 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.854537 15966 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: TxnStatusTablet state changed. Reason: New leader d9ba0462fec34779878c3967a4a02e72. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.854663 15966 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.855934 15964 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.856050 15964 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:50.856523 15964 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.856629 15964 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P d9ba0462fec34779878c3967a4a02e72: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:50.858374 15965 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.858482 15965 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P c65ca44e814e4dfa85c0e108ef27af99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.861001 15966 tablet_replica.cc:439] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9ba0462fec34779878c3967a4a02e72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 } } peers { permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 } } peers { permanent_uuid: "c65ca44e814e4dfa85c0e108ef27af99" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33427 } } }
I20260808 05:59:50.861112 15966 tablet_replica.cc:442] T 8d0451dabc4d4256a307c3826511cbb5 P a1679cd34a8b4af4a860cd6823d3cbb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:50.865768 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.861745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.13.229.67:33427)
0808 05:59:50.861875 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.861879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.861891 (+ 12us) server_negotiation.cc:407] Connection header received
0808 05:59:50.861960 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.861962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.862009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.862117 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:50.862375 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.863081 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.864262 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.864610 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.864717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.864921 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.865431 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.865546 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.865578 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.865635 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 05:59:50.876124 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.872636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:34799 (local address 127.13.229.65:44643)
0808 05:59:50.872747 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.872750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.872988 (+ 238us) server_negotiation.cc:407] Connection header received
0808 05:59:50.873106 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.873108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.873147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.873210 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:50.873461 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.873935 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.875098 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.875294 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.875376 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.875732 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.875792 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.875840 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.875982 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.876024 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 05:59:50.876214 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.872557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:44643 (local address 127.13.229.67:34799)
0808 05:59:50.872910 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.872924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.873025 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.873242 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.873245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.873252 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:50.873367 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.873374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.874037 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.874040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.874991 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.875003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.875399 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.875403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.875505 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.875560 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.875873 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.875880 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.875968 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.876049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
I20260808 05:59:50.882555 15583 raft_consensus.cc:1261] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Refusing update from remote peer c65ca44e814e4dfa85c0e108ef27af99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.882623 15737 raft_consensus.cc:1261] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Refusing update from remote peer c65ca44e814e4dfa85c0e108ef27af99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:50.883082 15991 consensus_queue.cc:1035] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9ba0462fec34779878c3967a4a02e72" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 44643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.883306 15965 consensus_queue.cc:1035] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1679cd34a8b4af4a860cd6823d3cbb4" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:50.907969 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.904095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:45359 (local address 127.13.229.67:33427)
0808 05:59:50.904199 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.904204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.904474 (+ 270us) server_negotiation.cc:407] Connection header received
0808 05:59:50.904663 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.904666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.904724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.904798 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:50.905186 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.905702 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.906688 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.907008 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.907081 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.907446 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.907523 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.907598 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.907722 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.907780 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 05:59:50.908337 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.904039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33427 (local address 127.13.229.65:45359)
0808 05:59:50.904396 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:50.904411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:50.904545 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:50.904924 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:50.904927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:50.904935 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:50.905038 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.905044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.905818 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.905820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.906554 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:50.906565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.907199 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.907202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:50.907272 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.907313 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:50.907603 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:50.907612 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 05:59:50.908121 (+ 509us) client_negotiation.cc:240] Negotiation successful
0808 05:59:50.908204 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260808 05:59:50.930668 15412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:50.930838 15412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:50.932807 15412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35a872f1797f4a6abe460e057cfccab8: Sending DeleteTablet for 3 replicas of tablet 522eb3ebe37848638c3e6c80a9cdf1bd
I20260808 05:59:50.933284 15562 tablet_service.cc:1514] Processing DeleteTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:60004
I20260808 05:59:50.933418 15715 tablet_service.cc:1514] Processing DeleteTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:46214
I20260808 05:59:50.933751 16027 tablet_replica.cc:330] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: stopping tablet replica
I20260808 05:59:50.933890 16027 raft_consensus.cc:2229] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:50.933872 15863 tablet_service.cc:1514] Processing DeleteTablet for tablet 522eb3ebe37848638c3e6c80a9cdf1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36576
I20260808 05:59:50.934006 16027 raft_consensus.cc:2258] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.934700 16027 ts_tablet_manager.cc:1905] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.935278 16028 tablet_replica.cc:330] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: stopping tablet replica
I20260808 05:59:50.935909 16028 raft_consensus.cc:2229] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 1 LEADER]: Raft consensus shutting down.
I20260808 05:59:50.936546 16028 raft_consensus.cc:2258] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.937194 16028 ts_tablet_manager.cc:1905] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.940184 16028 ts_tablet_manager.cc:1918] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:50.940328 16028 log.cc:1196] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/522eb3ebe37848638c3e6c80a9cdf1bd
I20260808 05:59:50.940670 16028 ts_tablet_manager.cc:1939] T 522eb3ebe37848638c3e6c80a9cdf1bd P c65ca44e814e4dfa85c0e108ef27af99: Deleting consensus metadata
I20260808 05:59:50.941366 15400 catalog_manager.cc:4841] TS c65ca44e814e4dfa85c0e108ef27af99 (127.13.229.67:33427): tablet 522eb3ebe37848638c3e6c80a9cdf1bd (table test-table [id=c668ad689a81442dacea8cd4d9a420f2]) successfully deleted
I20260808 05:59:50.941728 16026 tablet_replica.cc:330] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: stopping tablet replica
I20260808 05:59:50.941902 16026 raft_consensus.cc:2229] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:50.942059 16026 raft_consensus.cc:2258] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:50.942684 16026 ts_tablet_manager.cc:1905] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:50.945701 16027 ts_tablet_manager.cc:1918] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:50.945793 16027 log.cc:1196] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/522eb3ebe37848638c3e6c80a9cdf1bd
I20260808 05:59:50.946153 16027 ts_tablet_manager.cc:1939] T 522eb3ebe37848638c3e6c80a9cdf1bd P a1679cd34a8b4af4a860cd6823d3cbb4: Deleting consensus metadata
I20260808 05:59:50.946611 16026 ts_tablet_manager.cc:1918] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:50.946738 16026 log.cc:1196] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/522eb3ebe37848638c3e6c80a9cdf1bd
I20260808 05:59:50.947013 15397 catalog_manager.cc:4841] TS a1679cd34a8b4af4a860cd6823d3cbb4 (127.13.229.66:35713): tablet 522eb3ebe37848638c3e6c80a9cdf1bd (table test-table [id=c668ad689a81442dacea8cd4d9a420f2]) successfully deleted
I20260808 05:59:50.947089 16026 ts_tablet_manager.cc:1939] T 522eb3ebe37848638c3e6c80a9cdf1bd P d9ba0462fec34779878c3967a4a02e72: Deleting consensus metadata
I20260808 05:59:50.947798 15399 catalog_manager.cc:4841] TS d9ba0462fec34779878c3967a4a02e72 (127.13.229.65:44643): tablet 522eb3ebe37848638c3e6c80a9cdf1bd (table test-table [id=c668ad689a81442dacea8cd4d9a420f2]) successfully deleted
Aug 08 05:59:50 dist-test-slave-zkpd krb5kdc[15368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 05:59:50.958700 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.935675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.13.229.65:44643)
0808 05:59:50.935778 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.935782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.942549 (+ 6767us) server_negotiation.cc:407] Connection header received
0808 05:59:50.942746 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.942749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.942799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.942853 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.943853 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.944672 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.950368 (+ 5696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.950716 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.950823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.954899 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.954910 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.954925 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.954957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.956796 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.957253 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.957258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.957259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.957337 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.957635 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.957640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.957644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.957916 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.958190 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.958327 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.958458 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 05:59:50.959280 15626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33458
I20260808 05:59:50.969355 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:50.960936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.13.229.126:44515)
0808 05:59:50.961080 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:50.961085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:50.961480 (+ 395us) server_negotiation.cc:407] Connection header received
0808 05:59:50.961612 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:50.961616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:50.961669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:50.961752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:50.962404 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.962950 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.963764 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:50.964190 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:50.964301 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:50.966011 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:50.966015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:50.966017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:50.966045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:50.967607 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.968087 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.968091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.968093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.968188 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:50.968479 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:50.968483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:50.968485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:50.968671 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:50.968811 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:50.969063 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 05:59:50.969183 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 05:59:50.969820 15412 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49300
I20260808 05:59:50.971169 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15491
I20260808 05:59:50.982760 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15642
I20260808 05:59:50.994619 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15795
I20260808 05:59:51.002282 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 15381
2026-08-08T05:59:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2969 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:51 dist-test-slave-zkpd krb5kdc[16035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:51 dist-test-slave-zkpd krb5kdc[16035](info): set up 2 sockets
Aug 08 05:59:51 dist-test-slave-zkpd krb5kdc[16035](info): commencing operation
krb5kdc: starting...
W20260808 05:59:52.084393 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T05:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T05:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 05:59:52.208756 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41801
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:52.324908 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:52.325145 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:52.325183 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:52.329063 16049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 05:59:52.329113 16049 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 05:59:52.329129 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:52.329146 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:52.329162 16049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 05:59:52.329177 16049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 05:59:52.334223 16049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:41801
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:52.335124 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:52.336071 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:52.344180 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:52.344172 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:52.344172 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:52.344237 16049 server_base.cc:1034] running on GCE node
I20260808 05:59:52.344975 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:52.346014 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:52.347210 16049 hybrid_clock.cc:648] HybridClock initialized: now 1786168792347182 us; error 39 us; skew 500 ppm
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:52.350033 16049 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 05:59:52.350879 16049 webserver.cc:460] Webserver started at http://127.13.229.126:44263/ using document root <none> and password file <none>
I20260808 05:59:52.351428 16049 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:52.351513 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:52.351753 16049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:52.353917 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "f15c97afc69244cca157ac783f321da5"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "8a41abe3b400f2727513979b9a2f7c5d"
server_key_iv: "3540421cb2ca0124f338156f2b135407"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.354488 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "f15c97afc69244cca157ac783f321da5"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "8a41abe3b400f2727513979b9a2f7c5d"
server_key_iv: "3540421cb2ca0124f338156f2b135407"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.358165 16049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260808 05:59:52.360760 16066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:52.361950 16049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 05:59:52.362090 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "f15c97afc69244cca157ac783f321da5"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "8a41abe3b400f2727513979b9a2f7c5d"
server_key_iv: "3540421cb2ca0124f338156f2b135407"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.362198 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:52.380426 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:52.383286 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:52.383504 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:52.391415 16049 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:41801
I20260808 05:59:52.391446 16137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:41801 every 8 connection(s)
I20260808 05:59:52.392361 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 05:59:52.395116 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:52.395138 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16049
I20260808 05:59:52.395238 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 05:59:52.395535 14229 external_mini_cluster.cc:1428] Setting key a06b81c99e2ad8585f39bdb1b0055677
I20260808 05:59:52.401439 16138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Bootstrap starting.
I20260808 05:59:52.403697 16138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:52.404502 16138 log.cc:824] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Log is configured to *not* fsync() on all Append() calls
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:52.406766 16138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: No bootstrap required, opened a new log
I20260808 05:59:52.410017 16138 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } }
I20260808 05:59:52.410285 16138 raft_consensus.cc:374] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:52.410372 16138 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15c97afc69244cca157ac783f321da5, State: Initialized, Role: FOLLOWER
I20260808 05:59:52.410679 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.397004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.13.229.126:41801)
0808 05:59:52.397457 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:52.397477 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:52.397514 (+ 37us) server_negotiation.cc:407] Connection header received
0808 05:59:52.398413 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:52.398434 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:52.398780 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:52.399130 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:52.400045 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.401088 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.402094 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.402591 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.402784 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.405569 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:52.405575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:52.405593 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:52.405619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:52.407781 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.408259 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.408264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.408268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.408379 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.408659 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.408662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.408664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.409078 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:52.409293 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:52.409579 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 05:59:52.409843 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260808 05:59:52.410923 16138 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } }
I20260808 05:59:52.411082 16138 raft_consensus.cc:388] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 05:59:52.411154 16138 raft_consensus.cc:482] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 05:59:52.411267 16138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:52.412431 16138 raft_consensus.cc:504] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } }
I20260808 05:59:52.412797 16138 leader_election.cc:302] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f15c97afc69244cca157ac783f321da5; no voters:
I20260808 05:59:52.413167 16138 leader_election.cc:288] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 05:59:52.413286 16143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:52.413517 16143 raft_consensus.cc:686] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [term 1 LEADER]: Becoming Leader. State: Replica: f15c97afc69244cca157ac783f321da5, State: Running, Role: LEADER
I20260808 05:59:52.413836 16143 consensus_queue.cc:237] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } }
I20260808 05:59:52.414433 16138 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 05:59:52.415670 16144 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f15c97afc69244cca157ac783f321da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } } }
I20260808 05:59:52.415823 16144 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:52.416075 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f15c97afc69244cca157ac783f321da5. Latest consensus state: current_term: 1 leader_uuid: "f15c97afc69244cca157ac783f321da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15c97afc69244cca157ac783f321da5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41801 } } }
I20260808 05:59:52.416173 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5 [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:52.416323 16153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 05:59:52.419443 16153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 05:59:52.424903 16153 catalog_manager.cc:1347] Generated new cluster ID: 2d83d2cca8c2463cb589021986543a0d
I20260808 05:59:52.424973 16153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 05:59:52.438382 16153 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 05:59:52.439230 16153 catalog_manager.cc:1504] Loading token signing keys...
I20260808 05:59:52.446530 16153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Generated new TSK 0
I20260808 05:59:52.447180 16153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 05:59:52.485988 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:52.597728 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:52.597975 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:52.598021 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:52.601898 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:52.601948 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:52.601984 16165 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 05:59:52.601998 16165 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:52.602028 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 05:59:52.606969 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:52.607944 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:52.608920 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:52.618490 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:52.618503 16170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:52.618615 16165 server_base.cc:1034] running on GCE node
W20260808 05:59:52.618485 16171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:52.619167 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:52.619921 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:52.621107 16165 hybrid_clock.cc:648] HybridClock initialized: now 1786168792621078 us; error 44 us; skew 500 ppm
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:52.623853 16165 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 05:59:52.624715 16165 webserver.cc:460] Webserver started at http://127.13.229.65:38967/ using document root <none> and password file <none>
I20260808 05:59:52.625238 16165 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:52.625295 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:52.625466 16165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:52.627425 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "29bef97fa7b64b1eb87d42c859d1226e"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "120b7ac9b4de6cde46d876bf3c8e9cdf"
server_key_iv: "51b7d1a975030a3fc9631c52f8832a3b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.627990 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "29bef97fa7b64b1eb87d42c859d1226e"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "120b7ac9b4de6cde46d876bf3c8e9cdf"
server_key_iv: "51b7d1a975030a3fc9631c52f8832a3b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.631587 16165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 05:59:52.634074 16181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:52.635339 16165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:52.635468 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "29bef97fa7b64b1eb87d42c859d1226e"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "120b7ac9b4de6cde46d876bf3c8e9cdf"
server_key_iv: "51b7d1a975030a3fc9631c52f8832a3b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.635577 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:52.652074 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:52.655355 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:52.655524 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:52.656988 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:52.657066 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:52.657136 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:52.657181 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:52.668144 16165 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:42657
I20260808 05:59:52.668541 16306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:42657 every 8 connection(s)
I20260808 05:59:52.669153 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 05:59:52.671322 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.657877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:57527 (local address 127.13.229.126:41801)
0808 05:59:52.657998 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:52.658001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:52.659109 (+ 1108us) server_negotiation.cc:407] Connection header received
0808 05:59:52.660358 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:52.660362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:52.660436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:52.660542 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:52.661812 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.662434 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.663569 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.663823 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.663961 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.667512 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:52.667517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:52.667522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:52.667555 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:52.669327 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.669968 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.669972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.669974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.670069 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.670387 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.670390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.670391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.670574 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:52.670652 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:52.671093 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 05:59:52.671201 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 05:59:52.672149 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.658447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.65:57527)
0808 05:59:52.658959 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:52.659001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:52.660045 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:52.660753 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:52.660764 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:52.661165 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:52.661634 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.661648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.662560 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.662563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.663437 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.663447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.664015 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.664019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.664231 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.664833 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:52.664854 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:52.667338 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:52.669444 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.669450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.669465 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.669840 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.670166 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.670169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.670170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.670301 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.670661 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:52.670666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:52.670894 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 05:59:52.671045 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 05:59:52.671278 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260808 05:59:52.672523 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16165
I20260808 05:59:52.672665 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 05:59:52.673028 14229 external_mini_cluster.cc:1428] Setting key 382150e39ef446f46cf25c9516a4b6f5
I20260808 05:59:52.673578 16307 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41801
I20260808 05:59:52.673715 16307 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:52.674283 16307 heartbeater.cc:511] Master 127.13.229.126:41801 requested a full tablet report, sending...
I20260808 05:59:52.675786 16083 ts_manager.cc:194] Registered new tserver with Master: 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657)
I20260808 05:59:52.677052 16083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:57527
I20260808 05:59:52.684968 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.675801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.65:54509)
0808 05:59:52.676209 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:52.676224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:52.676303 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:52.678059 (+ 1756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:52.678065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:52.678303 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:52.678443 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.678452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.679475 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.679478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.680075 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.680083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.680699 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.680701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.680774 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.681290 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:52.681301 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:52.681593 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:52.683574 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.683576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.683578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.683767 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.684084 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.684087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.684088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.684163 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.684582 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:52.684584 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:52.684656 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 05:59:52.684734 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 05:59:52.684835 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260808 05:59:52.685144 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.675867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:54509 (local address 127.13.229.126:41801)
0808 05:59:52.677834 (+ 1967us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:52.677837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:52.677848 (+ 11us) server_negotiation.cc:407] Connection header received
0808 05:59:52.677896 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:52.677899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:52.677942 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:52.678023 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:52.678576 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.679308 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.680193 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.680580 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.680725 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.681732 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:52.681738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:52.681753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:52.681785 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:52.683468 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.683871 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.683874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.683877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.683995 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.684257 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.684261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.684265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.684488 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:52.684600 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:52.684832 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 05:59:52.684958 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895}
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 05:59:52.702534 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:52.812564 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:52.812844 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:52.812889 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:52.817023 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:52.817080 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:52.817147 16316 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 05:59:52.817185 16316 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:52.817265 16316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 05:59:52.822430 16316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:52.823459 16316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:52.824508 16316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:52.834213 16321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:52.834261 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:52.834254 16316 server_base.cc:1034] running on GCE node
W20260808 05:59:52.834285 16326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:52.834867 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:52.835412 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:52.836565 16316 hybrid_clock.cc:648] HybridClock initialized: now 1786168792836553 us; error 23 us; skew 500 ppm
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:52.839236 16316 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 05:59:52.840132 16316 webserver.cc:460] Webserver started at http://127.13.229.66:38337/ using document root <none> and password file <none>
I20260808 05:59:52.840647 16316 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:52.840705 16316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:52.840866 16316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:52.842800 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "ecb5dedb86154fd1a74197d896a36318"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "10421eb96dbace169216a3609cc6905b"
server_key_iv: "158c5993f5566d3e3c492adaf93d740b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.843298 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "ecb5dedb86154fd1a74197d896a36318"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "10421eb96dbace169216a3609cc6905b"
server_key_iv: "158c5993f5566d3e3c492adaf93d740b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.846887 16316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 05:59:52.849201 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:52.850548 16316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 05:59:52.850651 16316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "ecb5dedb86154fd1a74197d896a36318"
format_stamp: "Formatted at 2026-08-08 05:59:52 on dist-test-slave-zkpd"
server_key: "10421eb96dbace169216a3609cc6905b"
server_key_iv: "158c5993f5566d3e3c492adaf93d740b"
server_key_version: "encryptionkey@0"
I20260808 05:59:52.850775 16316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:52.862313 16316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:52.865710 16316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:52.865861 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:52.867439 16316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:52.867522 16316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:52.867573 16316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:52.867602 16316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:52 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:52.878603 16316 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:41171
I20260808 05:59:52.878691 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:41171 every 8 connection(s)
I20260808 05:59:52.879528 16316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 05:59:52.882149 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.868066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:36747 (local address 127.13.229.126:41801)
0808 05:59:52.868193 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:52.868198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:52.869796 (+ 1598us) server_negotiation.cc:407] Connection header received
0808 05:59:52.871057 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:52.871060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:52.871123 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:52.871200 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:52.872662 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.873585 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.875256 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.875474 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.875557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.878562 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:52.878566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:52.878568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:52.878600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:52.880233 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.880807 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.880810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.880811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.880905 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.881200 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.881202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.881203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.881377 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:52.881451 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:52.881931 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 05:59:52.882034 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 05:59:52.882934 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.869127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.66:36747)
0808 05:59:52.869628 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:52.869686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:52.870822 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:52.871367 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:52.871376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:52.871760 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:52.872479 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.872501 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.873737 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.873741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.875087 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.875112 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.875643 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.875648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.875926 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.876522 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:52.876548 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:52.878393 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:52.880344 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.880349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.880383 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.880702 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.880984 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.880986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.880988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.881116 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.881460 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:52.881464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:52.881683 (+ 219us) client_negotiation.cc:769] Sending connection context
0808 05:59:52.881866 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 05:59:52.882119 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260808 05:59:52.884117 16459 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41801
I20260808 05:59:52.884272 16459 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:52.884757 16459 heartbeater.cc:511] Master 127.13.229.126:41801 requested a full tablet report, sending...
I20260808 05:59:52.885792 16084 ts_manager.cc:194] Registered new tserver with Master: ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:52.886602 16084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:36747
I20260808 05:59:52.888805 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16316
I20260808 05:59:52.888947 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 05:59:52.889276 14229 external_mini_cluster.cc:1428] Setting key 3a6834934790e43cb83c894ab6ecba71
I20260808 05:59:52.894186 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.886408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:44337 (local address 127.13.229.126:41801)
0808 05:59:52.886522 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:52.886526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:52.886653 (+ 127us) server_negotiation.cc:407] Connection header received
0808 05:59:52.886766 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:52.886787 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:52.886831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:52.886932 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:52.887406 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.888201 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.889127 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.889448 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.889560 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.890791 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:52.890795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:52.890797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:52.890822 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:52.892455 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.892938 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.892943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.892948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.893048 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:52.893373 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:52.893375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:52.893377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:52.893590 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:52.893674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:52.893990 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 05:59:52.894082 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 05:59:52.894462 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:52.886282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.66:44337)
0808 05:59:52.886569 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:52.886584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:52.886686 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:52.886965 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:52.886968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:52.887175 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:52.887292 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.887299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.888346 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.888349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.888933 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:52.888945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:52.889556 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:52.889558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:52.889651 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:52.890188 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:52.890201 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:52.890642 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:52.892573 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.892576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.892577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.892815 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.893153 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:52.893156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:52.893160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:52.893266 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:52.893689 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:52.893694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:52.893824 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 05:59:52.893944 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 05:59:52.894101 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 05:59:52.919621 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:53.032624 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:53.032867 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:53.032979 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:53.036955 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:53.037009 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:53.037087 16470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 05:59:53.037123 16470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 05:59:53.037204 16470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 05:59:53.042549 16470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:53.043646 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:53.044705 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 05:59:53.054206 16470 server_base.cc:1034] running on GCE node
W20260808 05:59:53.054160 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:53.054160 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:53.054198 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:53.054842 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:53.055483 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:53.056635 16470 hybrid_clock.cc:648] HybridClock initialized: now 1786168793056620 us; error 25 us; skew 500 ppm
Aug 08 05:59:53 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168793, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:53.059273 16470 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 05:59:53.060159 16470 webserver.cc:460] Webserver started at http://127.13.229.67:38799/ using document root <none> and password file <none>
I20260808 05:59:53.060714 16470 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:53.060776 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:53.060979 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:53.062943 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204"
format_stamp: "Formatted at 2026-08-08 05:59:53 on dist-test-slave-zkpd"
server_key: "10c53867919940347b13bb8808c37215"
server_key_iv: "7ee026085daee692a7681ca45ab4f261"
server_key_version: "encryptionkey@0"
I20260808 05:59:53.063512 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204"
format_stamp: "Formatted at 2026-08-08 05:59:53 on dist-test-slave-zkpd"
server_key: "10c53867919940347b13bb8808c37215"
server_key_iv: "7ee026085daee692a7681ca45ab4f261"
server_key_version: "encryptionkey@0"
I20260808 05:59:53.067622 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 05:59:53.069964 16486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:53.071112 16470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 05:59:53.071231 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204"
format_stamp: "Formatted at 2026-08-08 05:59:53 on dist-test-slave-zkpd"
server_key: "10c53867919940347b13bb8808c37215"
server_key_iv: "7ee026085daee692a7681ca45ab4f261"
server_key_version: "encryptionkey@0"
I20260808 05:59:53.071331 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:53.095717 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:53.098563 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:53.098754 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:53.100298 16470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:53.100345 16470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:53.100415 16470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:53.100454 16470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 05:59:53 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168793, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:53.111752 16470 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38841
I20260808 05:59:53.112088 16608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38841 every 8 connection(s)
I20260808 05:59:53.112846 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 05:59:53.114723 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.101088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:58697 (local address 127.13.229.126:41801)
0808 05:59:53.101198 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.101202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.102344 (+ 1142us) server_negotiation.cc:407] Connection header received
0808 05:59:53.103486 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.103489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.103552 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.103635 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:53.105183 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.106075 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.107342 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.107589 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.107701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.110841 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:53.110844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:53.110846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:53.110873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:53.112538 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.113118 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.113120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.113121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.113222 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.113614 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.113618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.113620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.113885 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:53.114010 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.114499 (+ 489us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.114604 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 05:59:53.115631 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.101624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.67:58697)
0808 05:59:53.102162 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.102223 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.103226 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.103806 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.103817 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.104355 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:53.104991 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.105009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.106212 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.106215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.107209 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.107219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.107757 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.107761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.107993 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.108583 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:53.108602 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:53.110659 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:53.112650 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.112656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.112669 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.113019 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.113344 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.113347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.113349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.113514 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.114017 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:53.114055 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:53.114283 (+ 228us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.114443 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.114687 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":155,"threads_started":1}
I20260808 05:59:53.116168 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16470
I20260808 05:59:53.116304 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 05:59:53.116678 14229 external_mini_cluster.cc:1428] Setting key 3aef124dbbb36a1e513991a222e9583f
I20260808 05:59:53.117036 16609 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41801
I20260808 05:59:53.117189 16609 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:53.117990 16609 heartbeater.cc:511] Master 127.13.229.126:41801 requested a full tablet report, sending...
I20260808 05:59:53.118984 16083 ts_manager.cc:194] Registered new tserver with Master: 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841)
I20260808 05:59:53.119763 16083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:58697
I20260808 05:59:53.125670 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.119265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.13.229.67:58259)
0808 05:59:53.119548 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.119563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.119651 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.119861 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.119864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.120091 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:53.120213 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.120220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.121184 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.121187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.121729 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.121736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.122131 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.122134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.122203 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.122582 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:53.122593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:53.122900 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:53.124508 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.124510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.124512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.124685 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.124919 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.124921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.124922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.124991 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.125284 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:53.125286 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:53.125356 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.125433 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.125532 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260808 05:59:53.125675 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.119372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:58259 (local address 127.13.229.126:41801)
0808 05:59:53.119498 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.119501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.119613 (+ 112us) server_negotiation.cc:407] Connection header received
0808 05:59:53.119726 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.119728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.119761 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.119822 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:53.120335 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.121068 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.121829 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.122037 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.122116 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.123018 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:53.123021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:53.123023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:53.123043 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:53.124425 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.124763 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.124765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.124767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.124833 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.125062 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.125064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.125065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.125212 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:53.125284 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.125488 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.125573 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 05:59:53.130246 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 05:59:53 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 05:59:53 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:53.156461 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.148487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.13.229.126:41801)
0808 05:59:53.148617 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.148620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.148633 (+ 13us) server_negotiation.cc:407] Connection header received
0808 05:59:53.148674 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.148676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.148712 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.148779 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:53.149438 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.149939 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.150743 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.150963 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.151049 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.153412 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:53.153415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:53.153417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:53.153441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:53.155016 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.155361 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.155363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.155364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.155448 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.155665 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.155667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.155668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.155824 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:53.155974 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.156166 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.156256 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 05:59:53.159296 16083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 05:59:53.162257 16083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:53.176703 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.170448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:41171 (local address 127.0.0.1:36566)
0808 05:59:53.170863 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.170916 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.171051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.172101 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.172107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.172134 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:53.172272 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.172279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.173577 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.173584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.175018 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.175030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.176130 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.176239 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176286 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.176389 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.176509 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260808 05:59:53.176878 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.171427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.13.229.65:42657)
0808 05:59:53.171857 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.171880 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.171931 (+ 51us) server_negotiation.cc:407] Connection header received
0808 05:59:53.172007 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.172012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.172195 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.172397 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:53.172682 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.173801 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.175531 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.176318 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176426 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176551 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.176646 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.176731 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":58,"threads_started":1}
I20260808 05:59:53.177104 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.170612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.13.229.66:41171)
0808 05:59:53.171513 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.171528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.171556 (+ 28us) server_negotiation.cc:407] Connection header received
0808 05:59:53.171636 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.171641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.171867 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.172489 (+ 622us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:53.172632 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.173466 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.175132 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.175987 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176663 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176815 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.176862 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.176955 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":127,"threads_started":1}
I20260808 05:59:53.177280 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.170839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42657 (local address 127.0.0.1:39242)
0808 05:59:53.171083 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.171097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.171174 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.172369 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.172372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.172391 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:53.172500 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.172508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.173911 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.173914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.175406 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.175416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.176427 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.176522 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176541 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.176621 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.176672 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":53,"thread_start_us":90,"threads_started":1}
I20260808 05:59:53.178089 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.171416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.13.229.67:38841)
0808 05:59:53.172131 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.172143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.172163 (+ 20us) server_negotiation.cc:407] Connection header received
0808 05:59:53.172231 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.172249 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.172436 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.172566 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:53.172935 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.173991 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.175286 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.175985 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176096 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176223 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.177078 (+ 855us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.177180 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":110,"threads_started":1}
I20260808 05:59:53.178335 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.171202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38841 (local address 127.0.0.1:53878)
0808 05:59:53.171501 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.171517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.171603 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.172697 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.172700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.172718 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:53.172810 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.172817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.174095 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.174100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.175181 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.175191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.176300 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.176302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.176400 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.176426 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.176496 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.176564 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
I20260808 05:59:53.180075 16234 tablet_service.cc:1467] Processing CreateTablet for tablet e63ac8a147ac4072a5026581ce59d6da (DEFAULT_TABLE table=test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.180100 16387 tablet_service.cc:1467] Processing CreateTablet for tablet e63ac8a147ac4072a5026581ce59d6da (DEFAULT_TABLE table=test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.181066 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ac8a147ac4072a5026581ce59d6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.181121 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ac8a147ac4072a5026581ce59d6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.180830 16538 tablet_service.cc:1467] Processing CreateTablet for tablet e63ac8a147ac4072a5026581ce59d6da (DEFAULT_TABLE table=test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.181720 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e63ac8a147ac4072a5026581ce59d6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.187197 16633 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Bootstrap starting.
I20260808 05:59:53.188891 16634 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Bootstrap starting.
I20260808 05:59:53.189507 16635 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Bootstrap starting.
I20260808 05:59:53.190649 16633 tablet_bootstrap.cc:654] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.191015 16634 tablet_bootstrap.cc:654] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.191238 16635 tablet_bootstrap.cc:654] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.191942 16635 log.cc:824] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:53.191943 16634 log.cc:824] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:53.191953 16633 log.cc:824] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Log is configured to *not* fsync() on all Append() calls
I20260808 05:59:53.193667 16635 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: No bootstrap required, opened a new log
I20260808 05:59:53.193830 16633 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: No bootstrap required, opened a new log
I20260808 05:59:53.193902 16634 tablet_bootstrap.cc:492] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: No bootstrap required, opened a new log
I20260808 05:59:53.193894 16635 ts_tablet_manager.cc:1397] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 05:59:53.194016 16633 ts_tablet_manager.cc:1397] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 05:59:53.194043 16634 ts_tablet_manager.cc:1397] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 05:59:53.196745 16635 raft_consensus.cc:348] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.196974 16635 raft_consensus.cc:374] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.197031 16635 raft_consensus.cc:729] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.197463 16635 consensus_queue.cc:260] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.197840 16634 raft_consensus.cc:348] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.197840 16633 raft_consensus.cc:348] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.198074 16634 raft_consensus.cc:374] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.198074 16633 raft_consensus.cc:374] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.198124 16634 raft_consensus.cc:729] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.198134 16633 raft_consensus.cc:729] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bef97fa7b64b1eb87d42c859d1226e, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.198477 16635 ts_tablet_manager.cc:1428] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 05:59:53.198558 16634 consensus_queue.cc:260] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.198558 16633 consensus_queue.cc:260] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.198829 16459 heartbeater.cc:503] Master 127.13.229.126:41801 was elected leader, sending a full tablet report...
I20260808 05:59:53.199193 16307 heartbeater.cc:503] Master 127.13.229.126:41801 was elected leader, sending a full tablet report...
I20260808 05:59:53.199426 16633 ts_tablet_manager.cc:1428] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 05:59:53.199958 16609 heartbeater.cc:503] Master 127.13.229.126:41801 was elected leader, sending a full tablet report...
I20260808 05:59:53.200338 16634 ts_tablet_manager.cc:1428] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 05:59:53.363930 16610 tablet.cc:2369] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 05:59:53.380751 16460 tablet.cc:2369] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 05:59:53.390233 16641 raft_consensus.cc:482] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:53.390452 16641 raft_consensus.cc:504] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.391724 16641 leader_election.cc:288] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171), 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657)
I20260808 05:59:53.396119 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.392134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:41171 (local address 127.13.229.67:53581)
0808 05:59:53.392330 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.392346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.392462 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.392767 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.392770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.392789 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:53.392912 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.392918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.393932 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.393935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.395006 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.395015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.395667 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.395670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.395777 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.395804 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.395890 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.395993 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260808 05:59:53.396119 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.392120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:53581 (local address 127.13.229.66:41171)
0808 05:59:53.392251 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.392256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.392436 (+ 180us) server_negotiation.cc:407] Connection header received
0808 05:59:53.392585 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.392588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.392653 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.392751 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:53.393056 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.393818 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.395137 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.395545 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.395675 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.395732 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.395929 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.396002 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 05:59:53.396785 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ac8a147ac4072a5026581ce59d6da" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318" is_pre_election: true
I20260808 05:59:53.397109 16408 raft_consensus.cc:2454] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:53.397439 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.392120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:57229 (local address 127.13.229.65:42657)
0808 05:59:53.392251 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.392256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.392816 (+ 560us) server_negotiation.cc:407] Connection header received
0808 05:59:53.393011 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.393014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.393074 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.393157 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:53.393797 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.394571 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.396084 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.396717 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.396825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.396873 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.397236 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.397322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 05:59:53.397655 16489 leader_election.cc:302] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea9c3cb0c5a4e849b1a4312d77a7204, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:53.397871 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.392342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42657 (local address 127.13.229.67:57229)
0808 05:59:53.392722 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.392734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.392884 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.393170 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.393173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.393189 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:53.393662 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.393669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.394704 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.394707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.395937 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.395948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.396990 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.396993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.397087 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.397110 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.397573 (+ 463us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.397646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260808 05:59:53.398070 16641 raft_consensus.cc:2781] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:53.398154 16641 raft_consensus.cc:482] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:53.398202 16641 raft_consensus.cc:3037] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.398509 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ac8a147ac4072a5026581ce59d6da" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e" is_pre_election: true
I20260808 05:59:53.398770 16254 raft_consensus.cc:2454] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:53.399479 16641 raft_consensus.cc:504] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.399839 16641 leader_election.cc:288] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Requested vote from peers ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171), 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657)
I20260808 05:59:53.400166 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ac8a147ac4072a5026581ce59d6da" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318"
I20260808 05:59:53.400166 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e63ac8a147ac4072a5026581ce59d6da" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e"
I20260808 05:59:53.400288 16408 raft_consensus.cc:3037] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.400288 16254 raft_consensus.cc:3037] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.401556 16408 raft_consensus.cc:2454] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:53.401556 16254 raft_consensus.cc:2454] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:53.401883 16489 leader_election.cc:302] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea9c3cb0c5a4e849b1a4312d77a7204, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:53.402096 16641 raft_consensus.cc:2781] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:53.402307 16641 raft_consensus.cc:686] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Running, Role: LEADER
I20260808 05:59:53.402632 16641 consensus_queue.cc:237] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } }
I20260808 05:59:53.405915 16083 catalog_manager.cc:5495] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } health_report { overall_health: UNKNOWN } } }
W20260808 05:59:53.420565 16308 tablet.cc:2369] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 05:59:53 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168792, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:53.494408 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.486013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41801 (local address 127.0.0.1:47318)
0808 05:59:53.486351 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.486374 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.486503 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.486772 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.486774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.487192 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:53.487284 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.487290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.488104 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.488109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.488699 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.488707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.489326 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.489329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.489404 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.489953 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:53.489965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:53.491348 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:53.492904 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.492912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.492917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.493149 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.493406 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:53.493408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:53.493409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:53.493486 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:53.493873 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:53.493878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:53.493987 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.494094 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.494216 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260808 05:59:53.494426 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.486076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.13.229.126:41801)
0808 05:59:53.486215 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.486221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.486447 (+ 226us) server_negotiation.cc:407] Connection header received
0808 05:59:53.486605 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.486610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.486662 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.486747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:53.487414 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.488003 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.488975 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.489212 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.489322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.491482 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:53.491485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:53.491487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:53.491510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:53.492795 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.493238 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.493241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.493242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.493313 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:53.493565 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:53.493567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:53.493568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:53.493747 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:53.493832 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.494148 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.494271 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":30}
I20260808 05:59:53.498152 16083 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:47318:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 05:59:53.498878 16083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:53.503739 16538 tablet_service.cc:1467] Processing CreateTablet for tablet f8030df089af4d8bae5f90e0d9199a85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e36a7e4ddd3c442d8029f75776536314]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:53.503667 16234 tablet_service.cc:1467] Processing CreateTablet for tablet f8030df089af4d8bae5f90e0d9199a85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e36a7e4ddd3c442d8029f75776536314]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:53.503650 16387 tablet_service.cc:1467] Processing CreateTablet for tablet f8030df089af4d8bae5f90e0d9199a85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e36a7e4ddd3c442d8029f75776536314]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 05:59:53.504431 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8030df089af4d8bae5f90e0d9199a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.504431 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8030df089af4d8bae5f90e0d9199a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.504441 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8030df089af4d8bae5f90e0d9199a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.506834 16635 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: Bootstrap starting.
I20260808 05:59:53.507408 16634 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Bootstrap starting.
I20260808 05:59:53.507685 16633 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: Bootstrap starting.
I20260808 05:59:53.507777 16635 tablet_bootstrap.cc:654] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.508535 16633 tablet_bootstrap.cc:654] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.508610 16634 tablet_bootstrap.cc:654] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.509135 16635 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: No bootstrap required, opened a new log
I20260808 05:59:53.509231 16635 ts_tablet_manager.cc:1397] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:53.509728 16635 raft_consensus.cc:348] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.509850 16635 raft_consensus.cc:374] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.509868 16633 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: No bootstrap required, opened a new log
I20260808 05:59:53.509882 16634 tablet_bootstrap.cc:492] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: No bootstrap required, opened a new log
I20260808 05:59:53.509903 16635 raft_consensus.cc:729] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.509949 16633 ts_tablet_manager.cc:1397] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 05:59:53.509949 16634 ts_tablet_manager.cc:1397] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 05:59:53.510042 16635 consensus_queue.cc:260] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.510349 16635 ts_tablet_manager.cc:1428] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:53.510318 16639 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.510449 16639 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.510399 16633 raft_consensus.cc:348] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.510399 16634 raft_consensus.cc:348] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.510509 16633 raft_consensus.cc:374] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.510509 16634 raft_consensus.cc:374] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.510555 16633 raft_consensus.cc:729] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bef97fa7b64b1eb87d42c859d1226e, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.510555 16634 raft_consensus.cc:729] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.510679 16633 consensus_queue.cc:260] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.510689 16634 consensus_queue.cc:260] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.510962 16633 ts_tablet_manager.cc:1428] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:53.510962 16634 ts_tablet_manager.cc:1428] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 05:59:53.511036 16641 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.511159 16641 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.511513 16640 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.511642 16640 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.814184 16641 raft_consensus.cc:482] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:53.814407 16641 raft_consensus.cc:504] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.814904 16641 leader_election.cc:288] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:53.815423 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8030df089af4d8bae5f90e0d9199a85" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e" is_pre_election: true
I20260808 05:59:53.815423 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8030df089af4d8bae5f90e0d9199a85" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318" is_pre_election: true
I20260808 05:59:53.815586 16254 raft_consensus.cc:2454] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:53.815586 16408 raft_consensus.cc:2454] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:53.815873 16489 leader_election.cc:302] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea9c3cb0c5a4e849b1a4312d77a7204, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:53.816100 16641 raft_consensus.cc:2781] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:53.816169 16641 raft_consensus.cc:482] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:53.816226 16641 raft_consensus.cc:3037] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.817273 16641 raft_consensus.cc:504] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.817615 16641 leader_election.cc:288] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Requested vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:53.817977 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8030df089af4d8bae5f90e0d9199a85" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e"
I20260808 05:59:53.818015 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8030df089af4d8bae5f90e0d9199a85" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318"
I20260808 05:59:53.818127 16254 raft_consensus.cc:3037] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.818140 16408 raft_consensus.cc:3037] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:53.819041 16254 raft_consensus.cc:2454] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:53.819046 16408 raft_consensus.cc:2454] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:53.819309 16489 leader_election.cc:302] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea9c3cb0c5a4e849b1a4312d77a7204, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:53.819528 16641 raft_consensus.cc:2781] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:53.819646 16641 raft_consensus.cc:686] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Running, Role: LEADER
I20260808 05:59:53.819804 16641 consensus_queue.cc:237] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.820452 16644 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: TxnStatusTablet state changed. Reason: New leader 6ea9c3cb0c5a4e849b1a4312d77a7204. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.820566 16644 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:53.820834 16656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 05:59:53.821095 16656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 05:59:53.821514 16082 catalog_manager.cc:5495] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } health_report { overall_health: HEALTHY } } }
I20260808 05:59:53.865959 16644 consensus_queue.cc:1035] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:53.872867 16641 consensus_queue.cc:1035] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:53.886548 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.878197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.13.229.67:38841)
0808 05:59:53.882068 (+ 3871us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.882076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.882090 (+ 14us) server_negotiation.cc:407] Connection header received
0808 05:59:53.882156 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.882159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.882228 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.882350 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:53.882647 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.883472 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.884737 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.885127 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.885309 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.885623 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:53.886105 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:53.886210 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.886347 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.886403 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3797,"thread_start_us":85,"threads_started":1}
I20260808 05:59:53.886562 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.877934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38841 (local address 127.0.0.1:53886)
0808 05:59:53.878104 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.878123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.878224 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.882397 (+ 4173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.882400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.882410 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:53.882526 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.882532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.883587 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.883591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.884618 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.884630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.885320 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.885324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.885426 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.885482 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:53.886226 (+ 744us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:53.886246 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.886325 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.886424 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 05:59:53.891570 16408 raft_consensus.cc:1261] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:53.891649 16254 raft_consensus.cc:1261] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:53.892056 16641 consensus_queue.cc:1035] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:53.892231 16644 consensus_queue.cc:1035] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:53.894650 16639 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: TxnStatusTablet state changed. Reason: New leader 6ea9c3cb0c5a4e849b1a4312d77a7204. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.894763 16639 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.895149 16640 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: TxnStatusTablet state changed. Reason: New leader 6ea9c3cb0c5a4e849b1a4312d77a7204. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.895253 16640 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.898130 16644 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.898238 16644 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:53.898448 16644 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.898524 16644 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 6ea9c3cb0c5a4e849b1a4312d77a7204: This TxnStatusTablet replica's current role is: LEADER
I20260808 05:59:53.898764 16639 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.898855 16639 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P ecb5dedb86154fd1a74197d896a36318: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.899566 16640 tablet_replica.cc:439] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } } }
I20260808 05:59:53.899672 16640 tablet_replica.cc:442] T f8030df089af4d8bae5f90e0d9199a85 P 29bef97fa7b64b1eb87d42c859d1226e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 05:59:53.900527 16659 mvcc.cc:205] Tried to move back new op lower bound from 7316147379774853120 to 7316147379487260672. Current Snapshot: MvccSnapshot[applied={T|T < 7316147379774853120}]
I20260808 05:59:53.901419 16660 mvcc.cc:205] Tried to move back new op lower bound from 7316147379774853120 to 7316147379487260672. Current Snapshot: MvccSnapshot[applied={T|T < 7316147379774853120}]
I20260808 05:59:53.908695 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.905219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.13.229.67:38841)
0808 05:59:53.905361 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.905365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.905511 (+ 146us) server_negotiation.cc:407] Connection header received
0808 05:59:53.905631 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.905633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.905672 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.905735 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:53.905999 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.906486 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.907667 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.907880 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.908025 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.908285 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:53.908350 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:53.908404 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.908517 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.908561 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 05:59:53.919536 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.916066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38841 (local address 127.13.229.67:47021)
0808 05:59:53.916382 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:53.916396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:53.916513 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:53.916758 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:53.916760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:53.916766 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:53.916871 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.916876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.917737 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.917742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.918366 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:53.918375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.918828 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.918831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:53.918908 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.918954 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:53.919255 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:53.919263 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 05:59:53.919333 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 05:59:53.919412 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260808 05:59:53.919535 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:53.916117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:47021 (local address 127.13.229.67:38841)
0808 05:59:53.916241 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:53.916245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:53.916467 (+ 222us) server_negotiation.cc:407] Connection header received
0808 05:59:53.916627 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:53.916630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:53.916671 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:53.916750 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:53.916973 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.917642 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.918481 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:53.918705 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:53.918823 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:53.919100 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:53.919176 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:53.919252 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:53.919341 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 05:59:53.919387 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 05:59:53.964617 16082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:53.964838 16082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:53.967476 16082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Sending DeleteTablet for 3 replicas of tablet e63ac8a147ac4072a5026581ce59d6da
I20260808 05:59:53.968240 16387 tablet_service.cc:1514] Processing DeleteTablet for tablet e63ac8a147ac4072a5026581ce59d6da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36566
I20260808 05:59:53.968281 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet e63ac8a147ac4072a5026581ce59d6da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53878
I20260808 05:59:53.968883 16677 tablet_replica.cc:330] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: stopping tablet replica
I20260808 05:59:53.968919 16678 tablet_replica.cc:330] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: stopping tablet replica
I20260808 05:59:53.968866 16082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 05:59:53.969218 16678 raft_consensus.cc:2229] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Raft consensus shutting down.
W20260808 05:59:53.969293 16082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:53.969553 16678 raft_consensus.cc:2258] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:53.969551 16677 raft_consensus.cc:2229] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:53.969877 16677 raft_consensus.cc:2258] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:53.970093 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet e63ac8a147ac4072a5026581ce59d6da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:39242
I20260808 05:59:53.970770 16678 ts_tablet_manager.cc:1905] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:53.971320 16677 ts_tablet_manager.cc:1905] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:53.973918 16678 ts_tablet_manager.cc:1918] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:53.974020 16678 log.cc:1196] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/e63ac8a147ac4072a5026581ce59d6da
I20260808 05:59:53.974416 16678 ts_tablet_manager.cc:1939] T e63ac8a147ac4072a5026581ce59d6da P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting consensus metadata
I20260808 05:59:53.974448 16677 ts_tablet_manager.cc:1918] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:53.974999 16387 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0150589c294ad9b69e86877743dfd6 (DEFAULT_TABLE table=test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.975085 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0150589c294ad9b69e86877743dfd6 (DEFAULT_TABLE table=test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.975250 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0150589c294ad9b69e86877743dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.975334 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0150589c294ad9b69e86877743dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.976099 16679 tablet_replica.cc:330] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: stopping tablet replica
I20260808 05:59:53.976429 16679 raft_consensus.cc:2229] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:53.976643 16679 raft_consensus.cc:2258] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:53.977721 16635 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Bootstrap starting.
I20260808 05:59:53.978279 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0150589c294ad9b69e86877743dfd6 (DEFAULT_TABLE table=test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:53.978538 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0150589c294ad9b69e86877743dfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:53.978708 16635 tablet_bootstrap.cc:654] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.978726 16679 ts_tablet_manager.cc:1905] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:53.979348 16634 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Bootstrap starting.
I20260808 05:59:53.980235 16634 tablet_bootstrap.cc:654] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.981015 16633 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Bootstrap starting.
I20260808 05:59:53.981134 16677 log.cc:1196] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/e63ac8a147ac4072a5026581ce59d6da
I20260808 05:59:53.981549 16677 ts_tablet_manager.cc:1939] T e63ac8a147ac4072a5026581ce59d6da P ecb5dedb86154fd1a74197d896a36318: Deleting consensus metadata
I20260808 05:59:53.981946 16633 tablet_bootstrap.cc:654] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:53.982882 16070 catalog_manager.cc:4841] TS ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171): tablet e63ac8a147ac4072a5026581ce59d6da (table test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]) successfully deleted
I20260808 05:59:53.983168 16679 ts_tablet_manager.cc:1918] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:53.983268 16679 log.cc:1196] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/e63ac8a147ac4072a5026581ce59d6da
I20260808 05:59:53.983630 16679 ts_tablet_manager.cc:1939] T e63ac8a147ac4072a5026581ce59d6da P 29bef97fa7b64b1eb87d42c859d1226e: Deleting consensus metadata
I20260808 05:59:53.983944 16070 catalog_manager.cc:4841] TS 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841): tablet e63ac8a147ac4072a5026581ce59d6da (table test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]) successfully deleted
I20260808 05:59:53.984231 16635 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: No bootstrap required, opened a new log
I20260808 05:59:53.984321 16635 ts_tablet_manager.cc:1397] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 05:59:53.984756 16635 raft_consensus.cc:348] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.984822 16633 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: No bootstrap required, opened a new log
I20260808 05:59:53.984871 16635 raft_consensus.cc:374] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.984863 16068 catalog_manager.cc:4841] TS 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657): tablet e63ac8a147ac4072a5026581ce59d6da (table test-table [id=47b3b68a34d14b04ad1ee4ff9f9e0872]) successfully deleted
I20260808 05:59:53.984910 16635 raft_consensus.cc:729] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.984910 16633 ts_tablet_manager.cc:1397] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 05:59:53.985028 16635 consensus_queue.cc:260] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.985317 16635 ts_tablet_manager.cc:1428] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 05:59:53.985572 16633 raft_consensus.cc:348] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.985687 16633 raft_consensus.cc:374] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.985720 16633 raft_consensus.cc:729] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bef97fa7b64b1eb87d42c859d1226e, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.985829 16633 consensus_queue.cc:260] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.986045 16634 tablet_bootstrap.cc:492] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: No bootstrap required, opened a new log
I20260808 05:59:53.986064 16633 ts_tablet_manager.cc:1428] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:53.986125 16634 ts_tablet_manager.cc:1397] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260808 05:59:53.986557 16634 raft_consensus.cc:348] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.986660 16634 raft_consensus.cc:374] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:53.986693 16634 raft_consensus.cc:729] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Initialized, Role: FOLLOWER
I20260808 05:59:53.986805 16634 consensus_queue.cc:260] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:53.987085 16634 ts_tablet_manager.cc:1428] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:54.027230 16639 raft_consensus.cc:482] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:54.027424 16639 raft_consensus.cc:504] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:54.028661 16639 leader_election.cc:288] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841)
I20260808 05:59:54.033859 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.028832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42657 (local address 127.13.229.66:60533)
0808 05:59:54.029175 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:54.029191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:54.029301 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:54.030124 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:54.030130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:54.030179 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:54.030402 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.030412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.031665 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.031668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.032582 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.032592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.033392 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.033396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.033500 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.033528 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 05:59:54.033609 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 05:59:54.033679 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":73,"threads_started":1}
I20260808 05:59:54.033900 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.029010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:60533 (local address 127.13.229.65:42657)
0808 05:59:54.029606 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:54.029622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:54.029635 (+ 13us) server_negotiation.cc:407] Connection header received
0808 05:59:54.029692 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:54.029695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:54.029779 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:54.029853 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:54.030519 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.031531 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.032730 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.033257 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.033400 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.033456 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:54.033640 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 05:59:54.033742 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":62,"threads_started":1}
I20260808 05:59:54.034308 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0150589c294ad9b69e86877743dfd6" candidate_uuid: "ecb5dedb86154fd1a74197d896a36318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e" is_pre_election: true
I20260808 05:59:54.034472 16254 raft_consensus.cc:2454] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecb5dedb86154fd1a74197d896a36318 in term 0.
I20260808 05:59:54.034722 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.029337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:48751 (local address 127.13.229.67:38841)
0808 05:59:54.029536 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:54.029541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:54.029556 (+ 15us) server_negotiation.cc:407] Connection header received
0808 05:59:54.029636 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:54.029639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:54.029719 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:54.029895 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 05:59:54.030564 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.031430 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.033042 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.033629 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.033755 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.033811 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:54.034506 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 05:59:54.034588 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 05:59:54.034839 16336 leader_election.cc:302] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bef97fa7b64b1eb87d42c859d1226e, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:54.035107 16639 raft_consensus.cc:2781] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:54.035169 16639 raft_consensus.cc:482] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:54.035184 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.028938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38841 (local address 127.13.229.66:48751)
0808 05:59:54.029294 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:54.029307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:54.029385 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:54.030071 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:54.030075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:54.030096 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 05:59:54.030454 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.030463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.031546 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.031549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.032900 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.032912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.033874 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.033878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.034322 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.034343 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 05:59:54.034932 (+ 589us) client_negotiation.cc:240] Negotiation successful
0808 05:59:54.035013 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":118,"spinlock_wait_cycles":16128,"thread_start_us":95,"threads_started":1}
I20260808 05:59:54.035238 16639 raft_consensus.cc:3037] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:54.035756 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0150589c294ad9b69e86877743dfd6" candidate_uuid: "ecb5dedb86154fd1a74197d896a36318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" is_pre_election: true
I20260808 05:59:54.036005 16560 raft_consensus.cc:2454] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecb5dedb86154fd1a74197d896a36318 in term 0.
I20260808 05:59:54.036504 16639 raft_consensus.cc:504] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:54.036820 16639 leader_election.cc:288] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [CANDIDATE]: Term 1 election: Requested vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841)
I20260808 05:59:54.037132 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0150589c294ad9b69e86877743dfd6" candidate_uuid: "ecb5dedb86154fd1a74197d896a36318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e"
I20260808 05:59:54.037147 16560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0150589c294ad9b69e86877743dfd6" candidate_uuid: "ecb5dedb86154fd1a74197d896a36318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204"
I20260808 05:59:54.037267 16254 raft_consensus.cc:3037] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:54.037277 16560 raft_consensus.cc:3037] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:54.038252 16560 raft_consensus.cc:2454] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb5dedb86154fd1a74197d896a36318 in term 1.
I20260808 05:59:54.038398 16254 raft_consensus.cc:2454] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb5dedb86154fd1a74197d896a36318 in term 1.
I20260808 05:59:54.038617 16337 leader_election.cc:302] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ea9c3cb0c5a4e849b1a4312d77a7204, ecb5dedb86154fd1a74197d896a36318; no voters:
I20260808 05:59:54.038904 16639 raft_consensus.cc:2781] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:54.039157 16639 raft_consensus.cc:686] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 1 LEADER]: Becoming Leader. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Running, Role: LEADER
I20260808 05:59:54.039415 16639 consensus_queue.cc:237] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:54.041669 16082 catalog_manager.cc:5495] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecb5dedb86154fd1a74197d896a36318 (127.13.229.66). New cstate: current_term: 1 leader_uuid: "ecb5dedb86154fd1a74197d896a36318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } health_report { overall_health: UNKNOWN } } }
I20260808 05:59:54.068259 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.063643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.13.229.66:41171)
0808 05:59:54.064088 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:54.064095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:54.064108 (+ 13us) server_negotiation.cc:407] Connection header received
0808 05:59:54.064175 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:54.064182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:54.064237 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:54.064353 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:54.064642 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.065213 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.066413 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.066686 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.066769 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.067038 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:54.067557 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:54.067673 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:54.067794 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 05:59:54.067839 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":296,"threads_started":1}
I20260808 05:59:54.079329 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.075391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38841 (local address 127.13.229.66:56631)
0808 05:59:54.075871 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:54.075883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:54.076088 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:54.076342 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:54.076345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:54.076351 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:54.076445 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.076449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.077417 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.077419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.078084 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.078094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.078660 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.078662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.078733 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.078793 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:54.079086 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:54.079091 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 05:59:54.079142 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 05:59:54.079205 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":160,"threads_started":1}
I20260808 05:59:54.079347 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.075524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:56631 (local address 127.13.229.67:38841)
0808 05:59:54.075662 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:54.075667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:54.076015 (+ 348us) server_negotiation.cc:407] Connection header received
0808 05:59:54.076178 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:54.076182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:54.076242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:54.076322 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:54.076548 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.077300 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.078202 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.078547 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.078651 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.078920 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:54.079002 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:54.079076 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:54.079154 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 05:59:54.079205 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 05:59:54.085546 16254 raft_consensus.cc:1261] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Refusing update from remote peer ecb5dedb86154fd1a74197d896a36318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:54.085690 16560 raft_consensus.cc:1261] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Refusing update from remote peer ecb5dedb86154fd1a74197d896a36318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:54.086097 16686 consensus_queue.cc:1035] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:54.086267 16639 consensus_queue.cc:1035] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:54.093945 16662 mvcc.cc:205] Tried to move back new op lower bound from 7316147380569186304 to 7316147380387700736. Current Snapshot: MvccSnapshot[applied={T|T < 7316147380569186304}]
I20260808 05:59:54.107025 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.103512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:41171 (local address 127.13.229.67:55627)
0808 05:59:54.103646 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:54.103662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:54.103743 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:54.104137 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:54.104140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:54.104148 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 05:59:54.104254 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.104259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.105049 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.105054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.105875 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:54.105885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.106352 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.106356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:54.106442 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.106479 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 05:59:54.106752 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 05:59:54.106761 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 05:59:54.106828 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 05:59:54.106912 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 05:59:54.107025 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:54.103570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:55627 (local address 127.13.229.66:41171)
0808 05:59:54.103940 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:54.103945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:54.103957 (+ 12us) server_negotiation.cc:407] Connection header received
0808 05:59:54.104002 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:54.104005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:54.104044 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:54.104122 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:54.104398 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.104939 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.105994 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:54.106238 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:54.106349 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:54.106595 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:54.106673 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:54.106747 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:54.106842 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 05:59:54.106892 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
W20260808 05:59:54.107735 16660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 05:59:55.672403 16082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:55.672547 16082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:55.675065 16082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Sending DeleteTablet for 3 replicas of tablet 4c0150589c294ad9b69e86877743dfd6
I20260808 05:59:55.675624 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0150589c294ad9b69e86877743dfd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:39242
I20260808 05:59:55.675707 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0150589c294ad9b69e86877743dfd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53878
I20260808 05:59:55.676118 16722 tablet_replica.cc:330] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: stopping tablet replica
I20260808 05:59:55.676239 16722 raft_consensus.cc:2229] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:55.676865 16722 raft_consensus.cc:2258] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.677443 16722 ts_tablet_manager.cc:1905] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.680166 16722 ts_tablet_manager.cc:1918] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 05:59:55.680265 16722 log.cc:1196] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/4c0150589c294ad9b69e86877743dfd6
I20260808 05:59:55.680294 16387 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0150589c294ad9b69e86877743dfd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36566
I20260808 05:59:55.680573 16722 ts_tablet_manager.cc:1939] T 4c0150589c294ad9b69e86877743dfd6 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting consensus metadata
I20260808 05:59:55.680668 16724 tablet_replica.cc:330] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: stopping tablet replica
I20260808 05:59:55.680696 16721 tablet_replica.cc:330] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: stopping tablet replica
I20260808 05:59:55.680842 16724 raft_consensus.cc:2229] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 1 LEADER]: Raft consensus shutting down.
I20260808 05:59:55.680938 16721 raft_consensus.cc:2229] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:55.681111 16721 raft_consensus.cc:2258] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.681197 16724 raft_consensus.cc:2258] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.681337 16070 catalog_manager.cc:4841] TS 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841): tablet 4c0150589c294ad9b69e86877743dfd6 (table test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]) successfully deleted
I20260808 05:59:55.681804 16724 ts_tablet_manager.cc:1905] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.681844 16721 ts_tablet_manager.cc:1905] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.684037 16724 ts_tablet_manager.cc:1918] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 05:59:55.684172 16724 log.cc:1196] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/4c0150589c294ad9b69e86877743dfd6
I20260808 05:59:55.684540 16724 ts_tablet_manager.cc:1939] T 4c0150589c294ad9b69e86877743dfd6 P ecb5dedb86154fd1a74197d896a36318: Deleting consensus metadata
I20260808 05:59:55.684849 16721 ts_tablet_manager.cc:1918] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 05:59:55.684940 16721 log.cc:1196] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/4c0150589c294ad9b69e86877743dfd6
I20260808 05:59:55.685237 16721 ts_tablet_manager.cc:1939] T 4c0150589c294ad9b69e86877743dfd6 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting consensus metadata
I20260808 05:59:55.685336 16070 catalog_manager.cc:4841] TS ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171): tablet 4c0150589c294ad9b69e86877743dfd6 (table test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]) successfully deleted
I20260808 05:59:55.685984 16068 catalog_manager.cc:4841] TS 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657): tablet 4c0150589c294ad9b69e86877743dfd6 (table test-table [id=2d76b8aa611a44c0bd1b75883cb33df4]) successfully deleted
Aug 08 05:59:55 dist-test-slave-zkpd krb5kdc[16035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 05:59:55 dist-test-slave-zkpd krb5kdc[16035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:55.708959 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:55.697243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.13.229.126:41801)
0808 05:59:55.697581 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:55.697590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:55.697606 (+ 16us) server_negotiation.cc:407] Connection header received
0808 05:59:55.697735 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:55.697738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:55.697793 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:55.697886 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:55.698561 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:55.699112 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:55.699999 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:55.700241 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:55.700357 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:55.705917 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:55.705921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:55.705924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:55.705953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:55.707260 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:55.707678 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:55.707682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:55.707684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:55.707767 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:55.708079 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:55.708083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:55.708085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:55.708265 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:55.708420 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:55.708659 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 05:59:55.708787 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":119,"threads_started":1}
I20260808 05:59:55.711311 16082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 05:59:55.711748 16082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:55.716476 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 (DEFAULT_TABLE table=test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.716662 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 (DEFAULT_TABLE table=test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.716790 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8421d9c4cd47bf8570b13c389b5986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.716944 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8421d9c4cd47bf8570b13c389b5986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.719583 16387 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 (DEFAULT_TABLE table=test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.719746 16734 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Bootstrap starting.
I20260808 05:59:55.719875 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8421d9c4cd47bf8570b13c389b5986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.720691 16734 tablet_bootstrap.cc:654] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.722160 16734 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: No bootstrap required, opened a new log
I20260808 05:59:55.722255 16734 ts_tablet_manager.cc:1397] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 05:59:55.722599 16737 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Bootstrap starting.
I20260808 05:59:55.722715 16735 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Bootstrap starting.
I20260808 05:59:55.722725 16734 raft_consensus.cc:348] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.722844 16734 raft_consensus.cc:374] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.722927 16734 raft_consensus.cc:729] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bef97fa7b64b1eb87d42c859d1226e, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.723069 16734 consensus_queue.cc:260] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.723469 16734 ts_tablet_manager.cc:1428] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 05:59:55.723594 16737 tablet_bootstrap.cc:654] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.723729 16735 tablet_bootstrap.cc:654] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.724767 16737 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: No bootstrap required, opened a new log
I20260808 05:59:55.724855 16737 ts_tablet_manager.cc:1397] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:55.724995 16735 tablet_bootstrap.cc:492] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: No bootstrap required, opened a new log
I20260808 05:59:55.725066 16735 ts_tablet_manager.cc:1397] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:55.725277 16737 raft_consensus.cc:348] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.725390 16737 raft_consensus.cc:374] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.725430 16737 raft_consensus.cc:729] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.725425 16735 raft_consensus.cc:348] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.725526 16735 raft_consensus.cc:374] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.725556 16735 raft_consensus.cc:729] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.725559 16737 consensus_queue.cc:260] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.725764 16735 consensus_queue.cc:260] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.725833 16737 ts_tablet_manager.cc:1428] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 05:59:55.726058 16735 ts_tablet_manager.cc:1428] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:55.728967 16699 raft_consensus.cc:482] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:55.729081 16699 raft_consensus.cc:504] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.729404 16699 leader_election.cc:288] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:55.729699 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8421d9c4cd47bf8570b13c389b5986" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e" is_pre_election: true
I20260808 05:59:55.729698 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8421d9c4cd47bf8570b13c389b5986" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318" is_pre_election: true
I20260808 05:59:55.729801 16254 raft_consensus.cc:2454] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:55.729835 16408 raft_consensus.cc:2454] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:55.730082 16488 leader_election.cc:302] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bef97fa7b64b1eb87d42c859d1226e, 6ea9c3cb0c5a4e849b1a4312d77a7204; no voters:
I20260808 05:59:55.730304 16699 raft_consensus.cc:2781] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:55.730348 16699 raft_consensus.cc:482] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:55.730405 16699 raft_consensus.cc:3037] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.731194 16699 raft_consensus.cc:504] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.731514 16699 leader_election.cc:288] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Requested vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:55.731854 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8421d9c4cd47bf8570b13c389b5986" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318"
I20260808 05:59:55.731957 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8421d9c4cd47bf8570b13c389b5986" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e"
I20260808 05:59:55.732033 16408 raft_consensus.cc:3037] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.732069 16254 raft_consensus.cc:3037] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.732837 16254 raft_consensus.cc:2454] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:55.732952 16408 raft_consensus.cc:2454] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:55.733114 16488 leader_election.cc:302] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bef97fa7b64b1eb87d42c859d1226e, 6ea9c3cb0c5a4e849b1a4312d77a7204; no voters:
I20260808 05:59:55.733397 16699 raft_consensus.cc:2781] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:55.733508 16699 raft_consensus.cc:686] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Running, Role: LEADER
I20260808 05:59:55.733649 16699 consensus_queue.cc:237] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.735205 16084 catalog_manager.cc:5495] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } health_report { overall_health: HEALTHY } } }
I20260808 05:59:55.750595 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:55.747383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.13.229.67:38841)
0808 05:59:55.747664 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:55.747671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:55.747681 (+ 10us) server_negotiation.cc:407] Connection header received
0808 05:59:55.747717 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:55.747719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:55.747757 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:55.747845 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 05:59:55.748164 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:55.748689 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:55.749561 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:55.749794 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:55.749899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:55.750153 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 05:59:55.750220 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 05:59:55.750303 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:55.750401 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 05:59:55.750451 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260808 05:59:55.756402 16408 raft_consensus.cc:1261] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:55.756426 16254 raft_consensus.cc:1261] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:55.756757 16699 consensus_queue.cc:1035] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:55.756880 16675 consensus_queue.cc:1035] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:55.784288 16082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:55.784432 16082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:55.786260 16082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Sending DeleteTablet for 3 replicas of tablet 1e8421d9c4cd47bf8570b13c389b5986
I20260808 05:59:55.786859 16538 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53878
I20260808 05:59:55.786892 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:39242
I20260808 05:59:55.787037 16722 tablet_replica.cc:330] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: stopping tablet replica
I20260808 05:59:55.787070 16387 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8421d9c4cd47bf8570b13c389b5986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:55 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36566
I20260808 05:59:55.787177 16722 raft_consensus.cc:2229] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Raft consensus shutting down.
I20260808 05:59:55.787258 16724 tablet_replica.cc:330] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: stopping tablet replica
I20260808 05:59:55.787268 16721 tablet_replica.cc:330] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: stopping tablet replica
I20260808 05:59:55.787381 16724 raft_consensus.cc:2229] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:55.787388 16721 raft_consensus.cc:2229] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:55.787401 16722 raft_consensus.cc:2258] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.787480 16721 raft_consensus.cc:2258] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.787480 16724 raft_consensus.cc:2258] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:55.787811 16082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 05:59:55.788141 16722 ts_tablet_manager.cc:1905] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.788231 16724 ts_tablet_manager.cc:1905] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.788471 16721 ts_tablet_manager.cc:1905] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:55.790185 16722 ts_tablet_manager.cc:1918] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:55.790273 16722 log.cc:1196] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/1e8421d9c4cd47bf8570b13c389b5986
I20260808 05:59:55.790548 16722 ts_tablet_manager.cc:1939] T 1e8421d9c4cd47bf8570b13c389b5986 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting consensus metadata
I20260808 05:59:55.790830 16724 ts_tablet_manager.cc:1918] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:55.790886 16721 ts_tablet_manager.cc:1918] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 05:59:55.790922 16724 log.cc:1196] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/1e8421d9c4cd47bf8570b13c389b5986
I20260808 05:59:55.790947 16721 log.cc:1196] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/1e8421d9c4cd47bf8570b13c389b5986
I20260808 05:59:55.791262 16724 ts_tablet_manager.cc:1939] T 1e8421d9c4cd47bf8570b13c389b5986 P ecb5dedb86154fd1a74197d896a36318: Deleting consensus metadata
I20260808 05:59:55.791401 16721 ts_tablet_manager.cc:1939] T 1e8421d9c4cd47bf8570b13c389b5986 P 29bef97fa7b64b1eb87d42c859d1226e: Deleting consensus metadata
I20260808 05:59:55.791180 16070 catalog_manager.cc:4841] TS 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841): tablet 1e8421d9c4cd47bf8570b13c389b5986 (table test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]) successfully deleted
I20260808 05:59:55.791965 16068 catalog_manager.cc:4841] TS 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657): tablet 1e8421d9c4cd47bf8570b13c389b5986 (table test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]) successfully deleted
I20260808 05:59:55.792019 16070 catalog_manager.cc:4841] TS ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171): tablet 1e8421d9c4cd47bf8570b13c389b5986 (table test-table [id=e92d250ccf0443f7afa39d14fa3be9b9]) successfully deleted
W20260808 05:59:55.792156 16082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 05:59:55.796233 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 (DEFAULT_TABLE table=test-table [id=2f76d0c8de474b1793cc24c2a71ac3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.796308 16387 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 (DEFAULT_TABLE table=test-table [id=2f76d0c8de474b1793cc24c2a71ac3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.796504 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb145e562db4295b5888e9ba4b0d233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.796576 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb145e562db4295b5888e9ba4b0d233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.798239 16737 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Bootstrap starting.
I20260808 05:59:55.798420 16538 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 (DEFAULT_TABLE table=test-table [id=2f76d0c8de474b1793cc24c2a71ac3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 05:59:55.798633 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb145e562db4295b5888e9ba4b0d233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:55.799079 16737 tablet_bootstrap.cc:654] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.799612 16734 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: Bootstrap starting.
I20260808 05:59:55.800535 16734 tablet_bootstrap.cc:654] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.801059 16735 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Bootstrap starting.
I20260808 05:59:55.801499 16737 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: No bootstrap required, opened a new log
I20260808 05:59:55.801558 16737 ts_tablet_manager.cc:1397] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 05:59:55.801771 16734 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: No bootstrap required, opened a new log
I20260808 05:59:55.801855 16734 ts_tablet_manager.cc:1397] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:55.801903 16735 tablet_bootstrap.cc:654] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:55.801873 16737 raft_consensus.cc:348] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.801949 16737 raft_consensus.cc:374] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.801970 16737 raft_consensus.cc:729] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb5dedb86154fd1a74197d896a36318, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.802093 16737 consensus_queue.cc:260] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.802353 16737 ts_tablet_manager.cc:1428] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 05:59:55.802281 16734 raft_consensus.cc:348] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.802423 16734 raft_consensus.cc:374] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.802453 16734 raft_consensus.cc:729] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bef97fa7b64b1eb87d42c859d1226e, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.802568 16734 consensus_queue.cc:260] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.802847 16734 ts_tablet_manager.cc:1428] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 05:59:55.803714 16735 tablet_bootstrap.cc:492] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: No bootstrap required, opened a new log
I20260808 05:59:55.803807 16735 ts_tablet_manager.cc:1397] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 05:59:55.804224 16735 raft_consensus.cc:348] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.804345 16735 raft_consensus.cc:374] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:55.804392 16735 raft_consensus.cc:729] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Initialized, Role: FOLLOWER
I20260808 05:59:55.804510 16735 consensus_queue.cc:260] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.804786 16735 ts_tablet_manager.cc:1428] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 05:59:55.862596 16704 raft_consensus.cc:482] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:55.862720 16704 raft_consensus.cc:504] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.863173 16704 leader_election.cc:288] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:55.863577 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb145e562db4295b5888e9ba4b0d233" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e" is_pre_election: true
I20260808 05:59:55.863585 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb145e562db4295b5888e9ba4b0d233" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318" is_pre_election: true
I20260808 05:59:55.863732 16254 raft_consensus.cc:2454] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:55.863739 16408 raft_consensus.cc:2454] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 0.
I20260808 05:59:55.864051 16488 leader_election.cc:302] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bef97fa7b64b1eb87d42c859d1226e, 6ea9c3cb0c5a4e849b1a4312d77a7204; no voters:
I20260808 05:59:55.864266 16704 raft_consensus.cc:2781] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 05:59:55.864331 16704 raft_consensus.cc:482] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 05:59:55.864380 16704 raft_consensus.cc:3037] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.865404 16704 raft_consensus.cc:504] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.865742 16704 leader_election.cc:288] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Requested vote from peers 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657), ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171)
I20260808 05:59:55.866060 16254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb145e562db4295b5888e9ba4b0d233" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bef97fa7b64b1eb87d42c859d1226e"
I20260808 05:59:55.866140 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb145e562db4295b5888e9ba4b0d233" candidate_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb5dedb86154fd1a74197d896a36318"
I20260808 05:59:55.866219 16254 raft_consensus.cc:3037] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.866297 16408 raft_consensus.cc:3037] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:55.867168 16254 raft_consensus.cc:2454] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:55.867384 16408 raft_consensus.cc:2454] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ea9c3cb0c5a4e849b1a4312d77a7204 in term 1.
I20260808 05:59:55.867460 16488 leader_election.cc:302] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bef97fa7b64b1eb87d42c859d1226e, 6ea9c3cb0c5a4e849b1a4312d77a7204; no voters:
I20260808 05:59:55.867681 16704 raft_consensus.cc:2781] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:55.867786 16704 raft_consensus.cc:686] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Becoming Leader. State: Replica: 6ea9c3cb0c5a4e849b1a4312d77a7204, State: Running, Role: LEADER
I20260808 05:59:55.867969 16704 consensus_queue.cc:237] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } }
I20260808 05:59:55.869524 16084 catalog_manager.cc:5495] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ea9c3cb0c5a4e849b1a4312d77a7204" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38841 } health_report { overall_health: HEALTHY } } }
I20260808 05:59:55.886045 16408 raft_consensus.cc:1261] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:55.886081 16254 raft_consensus.cc:1261] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e [term 1 FOLLOWER]: Refusing update from remote peer 6ea9c3cb0c5a4e849b1a4312d77a7204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 05:59:55.886438 16751 consensus_queue.cc:1035] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bef97fa7b64b1eb87d42c859d1226e" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 05:59:55.886605 16704 consensus_queue.cc:1035] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb5dedb86154fd1a74197d896a36318" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 05:59:55.901264 16697 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 05:59:57.450604 16082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:57.450750 16082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 05:59:57.452972 16082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15c97afc69244cca157ac783f321da5: Sending DeleteTablet for 3 replicas of tablet 9bb145e562db4295b5888e9ba4b0d233
I20260808 05:59:57.453534 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:57 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:39242
I20260808 05:59:57.453550 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:57 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53878
I20260808 05:59:57.453591 16387 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb145e562db4295b5888e9ba4b0d233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 05:59:57 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36566
I20260808 05:59:57.453877 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16165
I20260808 05:59:57.453910 16777 tablet_replica.cc:330] T 9bb145e562db4295b5888e9ba4b0d233 P 29bef97fa7b64b1eb87d42c859d1226e: stopping tablet replica
I20260808 05:59:57.455989 16776 tablet_replica.cc:330] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: stopping tablet replica
I20260808 05:59:57.456147 16776 raft_consensus.cc:2229] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 LEADER]: Raft consensus shutting down.
I20260808 05:59:57.456236 16778 tablet_replica.cc:330] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: stopping tablet replica
I20260808 05:59:57.456362 16776 raft_consensus.cc:2258] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:57.456467 16778 raft_consensus.cc:2229] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 05:59:57.456652 16778 raft_consensus.cc:2258] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 05:59:57.457154 16776 ts_tablet_manager.cc:1905] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:57.460080 16776 ts_tablet_manager.cc:1918] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 05:59:57.460179 16778 ts_tablet_manager.cc:1905] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 05:59:57.460240 16776 log.cc:1196] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/9bb145e562db4295b5888e9ba4b0d233
I20260808 05:59:57.460577 16776 ts_tablet_manager.cc:1939] T 9bb145e562db4295b5888e9ba4b0d233 P 6ea9c3cb0c5a4e849b1a4312d77a7204: Deleting consensus metadata
I20260808 05:59:57.461459 16070 catalog_manager.cc:4841] TS 6ea9c3cb0c5a4e849b1a4312d77a7204 (127.13.229.67:38841): tablet 9bb145e562db4295b5888e9ba4b0d233 (table test-table [id=2f76d0c8de474b1793cc24c2a71ac3c5]) successfully deleted
I20260808 05:59:57.464071 16778 ts_tablet_manager.cc:1918] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 05:59:57.464187 16778 log.cc:1196] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/9bb145e562db4295b5888e9ba4b0d233
I20260808 05:59:57.464479 16778 ts_tablet_manager.cc:1939] T 9bb145e562db4295b5888e9ba4b0d233 P ecb5dedb86154fd1a74197d896a36318: Deleting consensus metadata
I20260808 05:59:57.465251 16070 catalog_manager.cc:4841] TS ecb5dedb86154fd1a74197d896a36318 (127.13.229.66:41171): tablet 9bb145e562db4295b5888e9ba4b0d233 (table test-table [id=2f76d0c8de474b1793cc24c2a71ac3c5]) successfully deleted
W20260808 05:59:57.466818 16068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:42657 (error 108)
I20260808 05:59:57.467082 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16316
I20260808 05:59:57.472316 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:57.468755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42657 (local address 127.0.0.1:39254)
0808 05:59:57.472124 (+ 3369us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:57.472212 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":72,"threads_started":1}
W20260808 05:59:57.472568 16068 catalog_manager.cc:4584] TS 29bef97fa7b64b1eb87d42c859d1226e (127.13.229.65:42657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bb145e562db4295b5888e9ba4b0d233: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42657: connect: Connection refused (error 111)
I20260808 05:59:57.476226 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16470
I20260808 05:59:57.485054 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16049
2026-08-08T05:59:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6481 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:57 dist-test-slave-zkpd krb5kdc[16783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:57 dist-test-slave-zkpd krb5kdc[16783](info): set up 2 sockets
Aug 08 05:59:57 dist-test-slave-zkpd krb5kdc[16783](info): commencing operation
krb5kdc: starting...
W20260808 05:59:58.544797 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 05:59:58 dist-test-slave-zkpd krb5kdc[16783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T05:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T05:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 05:59:58.670696 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42401
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:58.782299 16797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:58.782598 16797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:58.782667 16797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:58.786798 16797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 05:59:58.786896 16797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:58.786936 16797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:58.786988 16797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 05:59:58.787024 16797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 05:59:58.792375 16797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:58.793535 16797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:58.794654 16797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:58.803727 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:58.803754 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:58.803751 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:58.803820 16797 server_base.cc:1034] running on GCE node
I20260808 05:59:58.804637 16797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:58.805893 16797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:58.807102 16797 hybrid_clock.cc:648] HybridClock initialized: now 1786168798807093 us; error 41 us; skew 500 ppm
Aug 08 05:59:58 dist-test-slave-zkpd krb5kdc[16783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168798, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:58.810056 16797 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 05:59:58.810997 16797 webserver.cc:460] Webserver started at http://127.13.229.126:34325/ using document root <none> and password file <none>
I20260808 05:59:58.811621 16797 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:58.811717 16797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:58.812008 16797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:58.814205 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "e14a161093424932b1ab1c5253a7253f"
format_stamp: "Formatted at 2026-08-08 05:59:58 on dist-test-slave-zkpd"
server_key: "178296e6f76b58b900faf59cc9c10acc"
server_key_iv: "e42ef61035b1db6f5361618641399dcb"
server_key_version: "encryptionkey@0"
I20260808 05:59:58.814847 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "e14a161093424932b1ab1c5253a7253f"
format_stamp: "Formatted at 2026-08-08 05:59:58 on dist-test-slave-zkpd"
server_key: "178296e6f76b58b900faf59cc9c10acc"
server_key_iv: "e42ef61035b1db6f5361618641399dcb"
server_key_version: "encryptionkey@0"
I20260808 05:59:58.818866 16797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 05:59:58.821382 16814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:58.822577 16797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 05:59:58.822721 16797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "e14a161093424932b1ab1c5253a7253f"
format_stamp: "Formatted at 2026-08-08 05:59:58 on dist-test-slave-zkpd"
server_key: "178296e6f76b58b900faf59cc9c10acc"
server_key_iv: "e42ef61035b1db6f5361618641399dcb"
server_key_version: "encryptionkey@0"
I20260808 05:59:58.822834 16797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:58.861395 16797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:58.864300 16797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:58.864516 16797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:58.871618 16797 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:42401
I20260808 05:59:58.871634 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:42401 every 8 connection(s)
I20260808 05:59:58.872608 16797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 05:59:58.875301 16868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 05:59:58.877132 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16797
I20260808 05:59:58.877256 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 05:59:58.877528 14229 external_mini_cluster.cc:1428] Setting key 3da8bcccdd4172932ad0dfb6e3eb20e6
I20260808 05:59:58.882026 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f: Bootstrap starting.
I20260808 05:59:58.884547 16868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f: Neither blocks nor log segments found. Creating new log.
I20260808 05:59:58.885438 16868 log.cc:824] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f: Log is configured to *not* fsync() on all Append() calls
Aug 08 05:59:58 dist-test-slave-zkpd krb5kdc[16783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:58.887722 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f: No bootstrap required, opened a new log
I20260808 05:59:58.891767 16868 raft_consensus.cc:348] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } }
I20260808 05:59:58.892035 16868 raft_consensus.cc:374] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 05:59:58.892036 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:58.878800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.13.229.126:42401)
0808 05:59:58.879207 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:58.879226 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:58.879268 (+ 42us) server_negotiation.cc:407] Connection header received
0808 05:59:58.880128 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:58.880153 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:58.880570 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:58.880932 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:58.881576 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:58.883005 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:58.883835 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:58.884576 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:58.884754 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:58.887510 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:58.887515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:58.887528 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:58.887552 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:58.889161 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:58.889672 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:58.889676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:58.889680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:58.889784 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:58.890114 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:58.890116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:58.890117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:58.890486 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:58.890636 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:58.890950 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 05:59:58.891248 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":97,"threads_started":1}
I20260808 05:59:58.892115 16868 raft_consensus.cc:729] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14a161093424932b1ab1c5253a7253f, State: Initialized, Role: FOLLOWER
I20260808 05:59:58.892645 16868 consensus_queue.cc:260] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } }
I20260808 05:59:58.892825 16868 raft_consensus.cc:388] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 05:59:58.892889 16874 raft_consensus.cc:482] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 05:59:58.893013 16874 raft_consensus.cc:504] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } }
I20260808 05:59:58.893376 16868 raft_consensus.cc:482] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 05:59:58.893388 16874 leader_election.cc:302] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e14a161093424932b1ab1c5253a7253f; no voters:
I20260808 05:59:58.893476 16868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 0 FOLLOWER]: Advancing to term 1
I20260808 05:59:58.893821 16874 leader_election.cc:288] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260808 05:59:58.894632 16868 raft_consensus.cc:504] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } }
I20260808 05:59:58.894763 16868 leader_election.cc:302] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e14a161093424932b1ab1c5253a7253f; no voters:
I20260808 05:59:58.894793 16875 raft_consensus.cc:2741] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 05:59:58.894872 16868 leader_election.cc:288] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 05:59:58.894997 16874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 05:59:58.895191 16874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [term 1 LEADER]: Becoming Leader. State: Replica: e14a161093424932b1ab1c5253a7253f, State: Running, Role: LEADER
I20260808 05:59:58.895470 16874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } }
I20260808 05:59:58.896061 16868 sys_catalog.cc:564] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 05:59:58.897137 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e14a161093424932b1ab1c5253a7253f. Latest consensus state: current_term: 1 leader_uuid: "e14a161093424932b1ab1c5253a7253f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } } }
I20260808 05:59:58.897264 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:58.897540 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e14a161093424932b1ab1c5253a7253f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14a161093424932b1ab1c5253a7253f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42401 } } }
I20260808 05:59:58.897634 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f [sys.catalog]: This master's current role is: LEADER
I20260808 05:59:58.897785 16884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 05:59:58.900770 16884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 05:59:58.906418 16884 catalog_manager.cc:1347] Generated new cluster ID: 6bcfe1403b4b46ffa4c7089fa0a91e74
I20260808 05:59:58.906486 16884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 05:59:58.912420 16884 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 05:59:58.913548 16884 catalog_manager.cc:1504] Loading token signing keys...
I20260808 05:59:58.918159 16884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e14a161093424932b1ab1c5253a7253f: Generated new TSK 0
I20260808 05:59:58.918835 16884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 05:59:58.956094 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:59.068572 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:59.068810 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:59.068871 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:59.072777 16896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:59.072827 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:59.072880 16896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 05:59:59.077961 16896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:59.078929 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:59.079919 16896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:59.089481 16902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:59.089491 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:59.089529 16896 server_base.cc:1034] running on GCE node
W20260808 05:59:59.089490 16903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:59.090163 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:59.090749 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:59.091964 16896 hybrid_clock.cc:648] HybridClock initialized: now 1786168799091936 us; error 30 us; skew 500 ppm
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:59.095474 16896 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 05:59:59.096480 16896 webserver.cc:460] Webserver started at http://127.13.229.65:40489/ using document root <none> and password file <none>
I20260808 05:59:59.097080 16896 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:59.097172 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:59.097450 16896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:59.099761 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "6c9e82056d144be0b775a0c1357a96ef"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "8c463866dd592afc581eab2d86cdc424"
server_key_iv: "2ce92251db9c4c5ef3c661f8978d413f"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.100382 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "6c9e82056d144be0b775a0c1357a96ef"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "8c463866dd592afc581eab2d86cdc424"
server_key_iv: "2ce92251db9c4c5ef3c661f8978d413f"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.104401 16896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 05:59:59.106797 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.108052 16896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:59.108179 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "6c9e82056d144be0b775a0c1357a96ef"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "8c463866dd592afc581eab2d86cdc424"
server_key_iv: "2ce92251db9c4c5ef3c661f8978d413f"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.108273 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:59.126080 16896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:59.129629 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:59.129835 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:59.130429 16896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:59.131225 16896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:59.131273 16896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.131340 16896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:59.131376 16896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.140466 16896 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:43273
I20260808 05:59:59.140573 17029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:43273 every 8 connection(s)
I20260808 05:59:59.141434 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 05:59:59.142536 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 16896
I20260808 05:59:59.142647 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 05:59:59.142952 14229 external_mini_cluster.cc:1428] Setting key a66c124cf77300d672348107ace7ee0e
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 05:59:59.156206 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.143312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:43415 (local address 127.13.229.126:42401)
0808 05:59:59.143474 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:59.143479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:59.144461 (+ 982us) server_negotiation.cc:407] Connection header received
0808 05:59:59.145768 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:59.145773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:59.145837 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:59.145933 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:59.147189 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.148073 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.149201 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.149722 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.149879 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.152277 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:59.152284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:59.152291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:59.152328 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:59.154150 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.154648 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.154651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.154652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.154753 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.155095 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.155098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.155100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.155369 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:59.155462 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:59.155953 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 05:59:59.156079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 05:59:59.157159 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.143788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42401 (local address 127.13.229.65:43415)
0808 05:59:59.144318 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:59.144368 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:59.145550 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:59.146135 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:59.146143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:59.146507 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:59.147017 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.147033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.148215 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.148217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.149069 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.149079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.149900 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.149903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.150051 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.150634 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:59.150649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:59.152101 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:59.154281 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.154286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.154297 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.154549 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.154838 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.154841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.154842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.154968 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.155462 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:59.155467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:59.155690 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 05:59:59.155846 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 05:59:59.156148 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260808 05:59:59.158461 17030 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42401
I20260808 05:59:59.158705 17030 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:59.159191 17030 heartbeater.cc:511] Master 127.13.229.126:42401 requested a full tablet report, sending...
I20260808 05:59:59.160815 16831 ts_manager.cc:194] Registered new tserver with Master: 6c9e82056d144be0b775a0c1357a96ef (127.13.229.65:43273)
I20260808 05:59:59.162580 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:43415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 05:59:59.175325 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:59.286367 17035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:59.286609 17035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:59.286651 17035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:59.291527 17035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:59.291582 17035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:59.291666 17035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 05:59:59.297035 17035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:59.298027 17035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:59.299321 17035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:59.309230 17040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:59.309273 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:59.309259 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:59.309465 17035 server_base.cc:1034] running on GCE node
I20260808 05:59:59.309862 17035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:59.310487 17035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:59.311668 17035 hybrid_clock.cc:648] HybridClock initialized: now 1786168799311649 us; error 27 us; skew 500 ppm
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:59.314323 17035 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 05:59:59.315157 17035 webserver.cc:460] Webserver started at http://127.13.229.66:45185/ using document root <none> and password file <none>
I20260808 05:59:59.315691 17035 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:59.315778 17035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:59.316068 17035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:59.318025 17035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "fcb1b7770c34412f8f4aedb1a4524871"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "04eabc17bb7e6330bf5749692b9c62b4"
server_key_iv: "0288852631b3c6661fd0d3a9e3e2fe26"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.318586 17035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb1b7770c34412f8f4aedb1a4524871"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "04eabc17bb7e6330bf5749692b9c62b4"
server_key_iv: "0288852631b3c6661fd0d3a9e3e2fe26"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.322360 17035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 05:59:59.324806 17051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.326084 17035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 05:59:59.326225 17035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "fcb1b7770c34412f8f4aedb1a4524871"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "04eabc17bb7e6330bf5749692b9c62b4"
server_key_iv: "0288852631b3c6661fd0d3a9e3e2fe26"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.326357 17035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:59.345513 17035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:59.348675 17035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:59.348894 17035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:59.349500 17035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:59.350335 17035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:59.350410 17035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.350495 17035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:59.350540 17035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.359397 17035 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:46089
I20260808 05:59:59.359431 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:46089 every 8 connection(s)
I20260808 05:59:59.360301 17035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 05:59:59.362074 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17035
I20260808 05:59:59.362198 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 05:59:59.362481 14229 external_mini_cluster.cc:1428] Setting key 2ec0963d9154491a957d634301b6489e
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 05:59:59.374399 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.362088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:36793 (local address 127.13.229.126:42401)
0808 05:59:59.362269 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:59.362274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:59.363144 (+ 870us) server_negotiation.cc:407] Connection header received
0808 05:59:59.364506 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:59.364512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:59.364572 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:59.364660 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:59.365813 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.366339 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.367496 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.367972 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.368099 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.370410 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:59.370416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:59.370422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:59.370452 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:59.372156 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.372657 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.372664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.372666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.372762 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.373134 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.373138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.373140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.373412 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:59.373520 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:59.374164 (+ 644us) server_negotiation.cc:299] Negotiation successful
0808 05:59:59.374273 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 05:59:59.375217 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.362477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42401 (local address 127.13.229.66:36793)
0808 05:59:59.362994 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:59.363045 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:59.364298 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:59.364852 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:59.364859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:59.365201 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:59.365657 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.365670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.366465 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.366469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.367351 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.367360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.368118 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.368121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.368266 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.368848 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:59.368865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:59.370244 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:59.372289 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.372293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.372304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.372548 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.372880 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.372883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.372884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.373024 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.373553 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:59.373558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:59.373864 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 05:59:59.374042 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 05:59:59.374395 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 05:59:59.376416 17171 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42401
I20260808 05:59:59.376654 17171 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:59.377123 17171 heartbeater.cc:511] Master 127.13.229.126:42401 requested a full tablet report, sending...
I20260808 05:59:59.378132 16831 ts_manager.cc:194] Registered new tserver with Master: fcb1b7770c34412f8f4aedb1a4524871 (127.13.229.66:46089)
I20260808 05:59:59.378693 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:36793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 05:59:59.390722 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 05:59:59.509552 17176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 05:59:59.509771 17176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 05:59:59.509811 17176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 05:59:59.513949 17176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 05:59:59.513999 17176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 05:59:59.514089 17176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 05:59:59.519261 17176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 05:59:59.520249 17176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 05:59:59.521200 17176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 05:59:59.530604 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:59.530594 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 05:59:59.530596 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 05:59:59.530884 17176 server_base.cc:1034] running on GCE node
I20260808 05:59:59.531339 17176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 05:59:59.531980 17176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 05:59:59.533159 17176 hybrid_clock.cc:648] HybridClock initialized: now 1786168799533130 us; error 43 us; skew 500 ppm
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 05:59:59.535840 17176 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 05:59:59.536664 17176 webserver.cc:460] Webserver started at http://127.13.229.67:42851/ using document root <none> and password file <none>
I20260808 05:59:59.537226 17176 fs_manager.cc:359] Metadata directory not provided
I20260808 05:59:59.537321 17176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 05:59:59.537544 17176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 05:59:59.539529 17176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "6f95de16b886472f92a0c63445fabf41"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "10465f98f9244749dd3a7c6cb0b35117"
server_key_iv: "7828c671a2b49cc014cd13364a9697dc"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.540129 17176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "6f95de16b886472f92a0c63445fabf41"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "10465f98f9244749dd3a7c6cb0b35117"
server_key_iv: "7828c671a2b49cc014cd13364a9697dc"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.543672 17176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 05:59:59.546063 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.547226 17176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 05:59:59.547360 17176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "6f95de16b886472f92a0c63445fabf41"
format_stamp: "Formatted at 2026-08-08 05:59:59 on dist-test-slave-zkpd"
server_key: "10465f98f9244749dd3a7c6cb0b35117"
server_key_iv: "7828c671a2b49cc014cd13364a9697dc"
server_key_version: "encryptionkey@0"
I20260808 05:59:59.547465 17176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 05:59:59.557981 17176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 05:59:59.561147 17176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 05:59:59.561364 17176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 05:59:59.561971 17176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 05:59:59.562793 17176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 05:59:59.562860 17176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.562928 17176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 05:59:59.562968 17176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 05:59:59.571839 17176 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38663
I20260808 05:59:59.571950 17311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38663 every 8 connection(s)
I20260808 05:59:59.572820 17176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 05:59:59.578997 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17176
I20260808 05:59:59.579099 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 05:59:59.579391 14229 external_mini_cluster.cc:1428] Setting key 3a6c75b2d30e6d63f71056469a997b3d
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[16783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168799, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 05:59:59.587314 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.574743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:51547 (local address 127.13.229.126:42401)
0808 05:59:59.574890 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:59.574894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:59.575732 (+ 838us) server_negotiation.cc:407] Connection header received
0808 05:59:59.577085 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:59.577088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:59.577148 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:59.577248 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:59.578520 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.579230 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.580924 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.581182 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.581307 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.583922 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 05:59:59.583927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 05:59:59.583930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 05:59:59.583964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 05:59:59.585311 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.585835 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.585839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.585841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.585926 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 05:59:59.586267 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 05:59:59.586272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 05:59:59.586274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 05:59:59.586456 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 05:59:59.586561 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 05:59:59.587031 (+ 470us) server_negotiation.cc:299] Negotiation successful
0808 05:59:59.587151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 05:59:59.588327 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 05:59:59.575134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42401 (local address 127.13.229.67:51547)
0808 05:59:59.575581 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 05:59:59.575625 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 05:59:59.576873 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 05:59:59.577429 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 05:59:59.577436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 05:59:59.577794 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 05:59:59.578286 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.578299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.579367 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.579371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.580789 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 05:59:59.580806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 05:59:59.581350 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 05:59:59.581355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 05:59:59.581558 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 05:59:59.582121 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 05:59:59.582139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 05:59:59.583725 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 05:59:59.585455 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.585461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.585476 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.585730 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.586026 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 05:59:59.586029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 05:59:59.586031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 05:59:59.586174 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 05:59:59.586560 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 05:59:59.586567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 05:59:59.586803 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 05:59:59.586991 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 05:59:59.587300 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260808 05:59:59.589453 17312 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42401
I20260808 05:59:59.589684 17312 heartbeater.cc:464] Registering TS with master...
I20260808 05:59:59.590149 17312 heartbeater.cc:511] Master 127.13.229.126:42401 requested a full tablet report, sending...
I20260808 05:59:59.591153 16831 ts_manager.cc:194] Registered new tserver with Master: 6f95de16b886472f92a0c63445fabf41 (127.13.229.67:38663)
I20260808 05:59:59.591763 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:51547
I20260808 05:59:59.593218 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 05:59:59.607375 16872 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 05:59:59.605947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.13.229.126:42401)
0808 05:59:59.606090 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 05:59:59.606095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 05:59:59.606329 (+ 234us) server_negotiation.cc:407] Connection header received
0808 05:59:59.606469 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 05:59:59.606472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 05:59:59.606513 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 05:59:59.606600 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 05:59:59.607258 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 05:59:59.608526 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16896
I20260808 05:59:59.616225 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17035
I20260808 05:59:59.623323 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17176
I20260808 05:59:59.630393 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 16797
2026-08-08T05:59:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2143 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[17325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[17325](info): set up 2 sockets
Aug 08 05:59:59 dist-test-slave-zkpd krb5kdc[17325](info): commencing operation
krb5kdc: starting...
W20260808 06:00:00.701421 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:00 dist-test-slave-zkpd krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:00.826392 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34801
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:00.938526 17338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:00.938791 17338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:00.938881 17338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:00.942848 17338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:00.942905 17338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:00.942924 17338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:00.942991 17338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:00.943024 17338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:00.948174 17338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:00.949267 17338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:00.950316 17338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:00.958639 17346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:00.958639 17344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:00.958880 17338 server_base.cc:1034] running on GCE node
W20260808 06:00:00.958644 17343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:00.959609 17338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:00.960770 17338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:00.961962 17338 hybrid_clock.cc:648] HybridClock initialized: now 1786168800961930 us; error 46 us; skew 500 ppm
Aug 08 06:00:00 dist-test-slave-zkpd krb5kdc[17325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168800, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:00.969525 17338 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:00.970455 17338 webserver.cc:460] Webserver started at http://127.13.229.126:45341/ using document root <none> and password file <none>
I20260808 06:00:00.971019 17338 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:00.971112 17338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:00.971357 17338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:00.973520 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "ab110df567f14cb0b19f24017f6702a8"
format_stamp: "Formatted at 2026-08-08 06:00:00 on dist-test-slave-zkpd"
server_key: "ae3b467219ffcbeda69b6f42fff924fe"
server_key_iv: "ce162531ca1afcacf1009d040189aa65"
server_key_version: "encryptionkey@0"
I20260808 06:00:00.974113 17338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "ab110df567f14cb0b19f24017f6702a8"
format_stamp: "Formatted at 2026-08-08 06:00:00 on dist-test-slave-zkpd"
server_key: "ae3b467219ffcbeda69b6f42fff924fe"
server_key_iv: "ce162531ca1afcacf1009d040189aa65"
server_key_version: "encryptionkey@0"
I20260808 06:00:00.977802 17338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:00.980165 17354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:00.981329 17338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:00:00.981468 17338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "ab110df567f14cb0b19f24017f6702a8"
format_stamp: "Formatted at 2026-08-08 06:00:00 on dist-test-slave-zkpd"
server_key: "ae3b467219ffcbeda69b6f42fff924fe"
server_key_iv: "ce162531ca1afcacf1009d040189aa65"
server_key_version: "encryptionkey@0"
I20260808 06:00:00.981580 17338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:01.004498 17338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:01.007845 17338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:01.008070 17338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:01.015214 17338 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:34801
I20260808 06:00:01.015525 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:34801 every 8 connection(s)
I20260808 06:00:01.016209 17338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:01.019002 17408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:01.022835 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17338
I20260808 06:00:01.022969 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:01.023277 14229 external_mini_cluster.cc:1428] Setting key 84116c5833d5e1c78cb14568d5d30ed4
I20260808 06:00:01.025053 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: Bootstrap starting.
I20260808 06:00:01.027274 17408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:01.028086 17408 log.cc:824] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:01.030315 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: No bootstrap required, opened a new log
Aug 08 06:00:01 dist-test-slave-zkpd krb5kdc[17325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:01.034170 17408 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } }
I20260808 06:00:01.034437 17408 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:01.034521 17408 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab110df567f14cb0b19f24017f6702a8, State: Initialized, Role: FOLLOWER
I20260808 06:00:01.034963 17408 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } }
I20260808 06:00:01.035118 17408 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:01.035208 17408 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:01.035337 17408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:01.036540 17408 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } }
I20260808 06:00:01.036881 17408 leader_election.cc:302] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab110df567f14cb0b19f24017f6702a8; no voters:
I20260808 06:00:01.037281 17408 leader_election.cc:288] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:01.037447 17413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:01.037735 17413 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [term 1 LEADER]: Becoming Leader. State: Replica: ab110df567f14cb0b19f24017f6702a8, State: Running, Role: LEADER
I20260808 06:00:01.038095 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } }
I20260808 06:00:01.038538 17408 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:01.038868 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:01.024646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.13.229.126:34801)
0808 06:00:01.025088 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:01.025108 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:01.025173 (+ 65us) server_negotiation.cc:407] Connection header received
0808 06:00:01.025840 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:01.025858 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:01.026181 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:01.026524 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:01.027396 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:01.028558 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:01.029878 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:01.030343 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:01.030519 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:01.033308 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:01.033314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:01.033327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:01.033355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:01.035401 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:01.035943 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:01.035949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:01.035955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:01.036077 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:01.036405 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:01.036407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:01.036408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:01.036884 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:01.037076 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:01.037412 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 06:00:01.037926 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260808 06:00:01.039656 17414 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab110df567f14cb0b19f24017f6702a8. Latest consensus state: current_term: 1 leader_uuid: "ab110df567f14cb0b19f24017f6702a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } } }
I20260808 06:00:01.039798 17414 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:01.040508 17415 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab110df567f14cb0b19f24017f6702a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab110df567f14cb0b19f24017f6702a8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34801 } } }
I20260808 06:00:01.040592 17415 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:01.042382 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 06:00:01.043581 17428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:00:01.043674 17428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:00:01.043746 17422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:01.044485 17422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:01.050879 17422 catalog_manager.cc:1347] Generated new cluster ID: f48d9c08bf8049c98be33c6eaac04699
I20260808 06:00:01.050944 17422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:01.061653 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:01.056740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.13.229.126:34801)
0808 06:00:01.056854 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:01.056857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:01.057223 (+ 366us) server_negotiation.cc:407] Connection header received
0808 06:00:01.057438 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:01.057441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:01.057492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:01.057573 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:01.058075 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:01.058631 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:01.059497 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:01.059757 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:01.059875 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:01.060775 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:01.060779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:01.060790 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:01.061207 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:01.061236 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:01.061252 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:01.061359 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:01.061442 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 06:00:01.061504 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:00:01.089432 17422 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:01.090426 17422 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:01.096437 17422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab110df567f14cb0b19f24017f6702a8: Generated new TSK 0
I20260808 06:00:01.097136 17422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:01.106371 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17338
2026-08-08T06:00:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1475 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:01 dist-test-slave-zkpd krb5kdc[17443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:01 dist-test-slave-zkpd krb5kdc[17443](info): set up 2 sockets
Aug 08 06:00:01 dist-test-slave-zkpd krb5kdc[17443](info): commencing operation
krb5kdc: starting...
W20260808 06:00:02.178445 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:02.310799 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:35089
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:02.425717 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:02.425992 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:02.426031 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:02.430085 17456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:02.430142 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:02.430162 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:02.430217 17456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:02.430251 17456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:02.435518 17456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:35089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:02.436666 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:02.437733 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:02.446189 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:02.446188 17463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:02.446192 17462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:02.446378 17456 server_base.cc:1034] running on GCE node
I20260808 06:00:02.447100 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:02.448323 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:02.449546 17456 hybrid_clock.cc:648] HybridClock initialized: now 1786168802449513 us; error 42 us; skew 500 ppm
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:02.452841 17456 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:02.453747 17456 webserver.cc:460] Webserver started at http://127.13.229.126:33889/ using document root <none> and password file <none>
I20260808 06:00:02.454296 17456 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:02.454360 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:02.454538 17456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:02.456790 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "0be0ce1610a44741b5d0328d6ffe54d1"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "d162f1bc4de0ac350921d12b3b8ef4ec"
server_key_iv: "c5bd59ac02194e2cffdd9f3fbd015360"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.457412 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "0be0ce1610a44741b5d0328d6ffe54d1"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "d162f1bc4de0ac350921d12b3b8ef4ec"
server_key_iv: "c5bd59ac02194e2cffdd9f3fbd015360"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.461581 17456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:00:02.464346 17472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.465811 17456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:00:02.465929 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "0be0ce1610a44741b5d0328d6ffe54d1"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "d162f1bc4de0ac350921d12b3b8ef4ec"
server_key_iv: "c5bd59ac02194e2cffdd9f3fbd015360"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.466027 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:02.496973 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:02.500862 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:02.501041 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:02.508457 17456 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:35089
I20260808 06:00:02.508741 17526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:35089 every 8 connection(s)
I20260808 06:00:02.509549 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:02.512444 17527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:02.517917 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1: Bootstrap starting.
I20260808 06:00:02.518548 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17456
I20260808 06:00:02.518649 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:02.518954 14229 external_mini_cluster.cc:1428] Setting key fb48db9667ca861f230bfb0111a4dec6
I20260808 06:00:02.520704 17527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:02.521446 17527 log.cc:824] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:02.523306 17527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1: No bootstrap required, opened a new log
I20260808 06:00:02.526693 17527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } }
I20260808 06:00:02.526947 17527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:02.527007 17527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be0ce1610a44741b5d0328d6ffe54d1, State: Initialized, Role: FOLLOWER
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:02.527487 17527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } }
I20260808 06:00:02.527688 17527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:02.527772 17527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:02.527874 17527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:02.529121 17527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } }
I20260808 06:00:02.529523 17527 leader_election.cc:302] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0be0ce1610a44741b5d0328d6ffe54d1; no voters:
I20260808 06:00:02.529891 17527 leader_election.cc:288] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:02.530005 17533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:02.530241 17533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [term 1 LEADER]: Becoming Leader. State: Replica: 0be0ce1610a44741b5d0328d6ffe54d1, State: Running, Role: LEADER
I20260808 06:00:02.530548 17533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } }
I20260808 06:00:02.531288 17527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:02.532254 17535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0be0ce1610a44741b5d0328d6ffe54d1. Latest consensus state: current_term: 1 leader_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } } }
I20260808 06:00:02.532232 17534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be0ce1610a44741b5d0328d6ffe54d1" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35089 } } }
I20260808 06:00:02.532385 17534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:02.532385 17535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:02.532996 17542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:02.534946 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:02.520199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.13.229.126:35089)
0808 06:00:02.520618 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:02.520634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:02.520670 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:00:02.521334 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:02.521351 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:02.521732 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:02.522066 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:02.522902 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.523982 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.524854 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.525357 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.525530 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:02.528207 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:02.528215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:02.528231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:02.528263 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:02.530496 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:02.531053 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:02.531058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:02.531065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:02.531202 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:02.531538 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:02.531541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:02.531542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:02.532136 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:02.533412 (+ 1276us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:02.533605 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 06:00:02.534048 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":155,"threads_started":1}
I20260808 06:00:02.536720 17542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:02.542145 17542 catalog_manager.cc:1347] Generated new cluster ID: cde95938ecb748febd40a5c516577ef0
I20260808 06:00:02.542223 17542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:02.548637 17542 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:02.549515 17542 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:02.552997 17542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0be0ce1610a44741b5d0328d6ffe54d1: Generated new TSK 0
I20260808 06:00:02.553646 17542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:02.583511 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:02.696591 17555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:02.696846 17555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:02.696894 17555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:02.700984 17555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:02.701037 17555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:02.701123 17555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:02.706344 17555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:02.707449 17555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:02.708534 17555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:02.718334 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:02.718328 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:02.718433 17555 server_base.cc:1034] running on GCE node
W20260808 06:00:02.718333 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:02.719059 17555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:02.719708 17555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:02.720890 17555 hybrid_clock.cc:648] HybridClock initialized: now 1786168802720884 us; error 55 us; skew 500 ppm
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:02.723613 17555 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:02.724542 17555 webserver.cc:460] Webserver started at http://127.13.229.65:40811/ using document root <none> and password file <none>
I20260808 06:00:02.725103 17555 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:02.725167 17555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:02.725407 17555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:02.727500 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "73315c10ef044d1daff1ba4fefd88758"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "db916b360e4c2464d42b6a48fd7678f6"
server_key_iv: "0039d9518790ecb1cffa4e703b1fe2d7"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.728123 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "73315c10ef044d1daff1ba4fefd88758"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "db916b360e4c2464d42b6a48fd7678f6"
server_key_iv: "0039d9518790ecb1cffa4e703b1fe2d7"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.732054 17555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:00:02.734550 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.735987 17555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:02.736117 17555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "73315c10ef044d1daff1ba4fefd88758"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "db916b360e4c2464d42b6a48fd7678f6"
server_key_iv: "0039d9518790ecb1cffa4e703b1fe2d7"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.736225 17555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:02.745756 17555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:02.749002 17555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:02.749209 17555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:02.749801 17555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:02.750610 17555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:02.750658 17555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.750725 17555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:02.750757 17555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.759773 17555 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:45485
I20260808 06:00:02.759795 17691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:45485 every 8 connection(s)
I20260808 06:00:02.760739 17555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:02.770630 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17555
I20260808 06:00:02.770772 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:02.771127 14229 external_mini_cluster.cc:1428] Setting key f1bb411c24660e4efe014062d75c52dc
I20260808 06:00:02.775031 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:02.762579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40463 (local address 127.13.229.126:35089)
0808 06:00:02.762722 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:02.762725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:02.763568 (+ 843us) server_negotiation.cc:407] Connection header received
0808 06:00:02.764969 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:02.764973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:02.765033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:02.765138 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:02.766292 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.766833 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.768028 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.768292 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.768408 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:02.771055 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:02.771060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:02.771062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:02.771094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:02.772723 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:02.773346 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:02.773352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:02.773356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:02.773471 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:02.773848 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:02.773854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:02.773859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:02.774119 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:02.774226 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:02.774705 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 06:00:02.774855 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:00:02.776283 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:02.762955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35089 (local address 127.13.229.65:40463)
0808 06:00:02.763413 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:02.763455 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:02.764746 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:02.765324 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:02.765331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:02.765680 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:02.766145 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:02.766157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.766965 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.766973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:02.767848 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:02.767857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:02.768450 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:02.768455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:02.768631 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:02.769190 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:02.769207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:02.770872 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:02.772868 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:02.772874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:02.772890 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:02.773237 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:02.773569 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:02.773571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:02.773573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:02.773707 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:02.774223 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:02.774228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:02.774465 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 06:00:02.774626 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 06:00:02.775059 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260808 06:00:02.777606 17692 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35089
I20260808 06:00:02.777900 17692 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:02.778541 17692 heartbeater.cc:511] Master 127.13.229.126:35089 requested a full tablet report, sending...
I20260808 06:00:02.780416 17489 ts_manager.cc:194] Registered new tserver with Master: 73315c10ef044d1daff1ba4fefd88758 (127.13.229.65:45485)
I20260808 06:00:02.782114 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:40463
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:02.800889 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:02.915236 17698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:02.915527 17698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:02.915613 17698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:02.919674 17698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:02.919775 17698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:02.919870 17698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:02.925232 17698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:02.926338 17698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:02.927448 17698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:02.937561 17706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:02.937572 17704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:02.937619 17703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:02.937736 17698 server_base.cc:1034] running on GCE node
I20260808 06:00:02.938282 17698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:02.938935 17698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:02.940145 17698 hybrid_clock.cc:648] HybridClock initialized: now 1786168802940098 us; error 32 us; skew 500 ppm
Aug 08 06:00:02 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:02.943131 17698 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:02.944169 17698 webserver.cc:460] Webserver started at http://127.13.229.66:41019/ using document root <none> and password file <none>
I20260808 06:00:02.944809 17698 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:02.944917 17698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:02.945176 17698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:02.947440 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "d77385822e6d45b5bfd3f5274b8e7939"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "4b79bdd10cecbc6ba5aae792a61358c7"
server_key_iv: "44982f71179209ee3df26d8840a12a76"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.948149 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "d77385822e6d45b5bfd3f5274b8e7939"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "4b79bdd10cecbc6ba5aae792a61358c7"
server_key_iv: "44982f71179209ee3df26d8840a12a76"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.952384 17698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 06:00:02.955039 17713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.956524 17698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:02.956673 17698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "d77385822e6d45b5bfd3f5274b8e7939"
format_stamp: "Formatted at 2026-08-08 06:00:02 on dist-test-slave-zkpd"
server_key: "4b79bdd10cecbc6ba5aae792a61358c7"
server_key_iv: "44982f71179209ee3df26d8840a12a76"
server_key_version: "encryptionkey@0"
I20260808 06:00:02.956808 17698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:02.990146 17698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:02.993922 17698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:02.994159 17698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:02.994853 17698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:02.995777 17698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:02.995855 17698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:02.995961 17698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:02.996012 17698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:03.005641 17698 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:32773
I20260808 06:00:03.005669 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:32773 every 8 connection(s)
I20260808 06:00:03.006637 17698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:03.008535 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17698
I20260808 06:00:03.008669 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:03.008975 14229 external_mini_cluster.cc:1428] Setting key 615397fb26c696418f80cdb88c3972ed
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168802, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:00:03.021832 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:03.008714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:59325 (local address 127.13.229.126:35089)
0808 06:00:03.008886 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:03.008891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:03.009857 (+ 966us) server_negotiation.cc:407] Connection header received
0808 06:00:03.011266 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:03.011269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:03.011320 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:03.011400 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:03.013001 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.013587 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.014829 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.015301 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.015439 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:03.017880 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:03.017885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:03.017889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:03.017926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:03.019627 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.020242 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.020251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.020252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.020332 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.020639 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.020642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.020643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.020872 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:03.020952 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:03.021554 (+ 602us) server_negotiation.cc:299] Negotiation successful
0808 06:00:03.021684 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:00:03.022675 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:03.009185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35089 (local address 127.13.229.66:59325)
0808 06:00:03.009680 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:03.009750 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:03.011055 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:03.011595 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:03.011603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:03.012065 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:03.012731 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:03.012753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.013740 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.013746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:03.014685 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:03.014695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.015489 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.015493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:03.015630 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:03.016265 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:03.016282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:03.017703 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:03.019753 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:03.019758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:03.019770 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:03.020144 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:03.020418 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:03.020420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:03.020422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:03.020551 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:03.020980 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:03.020986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:03.021217 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 06:00:03.021373 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 06:00:03.021627 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260808 06:00:03.024067 17831 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35089
I20260808 06:00:03.024356 17831 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:03.025048 17831 heartbeater.cc:511] Master 127.13.229.126:35089 requested a full tablet report, sending...
I20260808 06:00:03.026367 17489 ts_manager.cc:194] Registered new tserver with Master: d77385822e6d45b5bfd3f5274b8e7939 (127.13.229.66:32773)
I20260808 06:00:03.026990 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:59325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:03.040954 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:03.159446 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:03.159693 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:03.159735 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:03.163831 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:03.163913 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:03.164016 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:03.169133 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:03.170238 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:03.171317 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:03.180894 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:03.180939 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:03.180896 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:03.181095 17836 server_base.cc:1034] running on GCE node
I20260808 06:00:03.181576 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:03.182217 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:03.183380 17836 hybrid_clock.cc:648] HybridClock initialized: now 1786168803183344 us; error 47 us; skew 500 ppm
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168803, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:03.186189 17836 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:03.187081 17836 webserver.cc:460] Webserver started at http://127.13.229.67:34199/ using document root <none> and password file <none>
I20260808 06:00:03.187668 17836 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:03.187774 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:03.188060 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:03.190066 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "c5c79d04ea0e4ad68fdbc6b52fb1d0d0"
format_stamp: "Formatted at 2026-08-08 06:00:03 on dist-test-slave-zkpd"
server_key: "ca38b108f18be6809292219cea954aa9"
server_key_iv: "6c2ecc7904fe73d89757c61cb556ac68"
server_key_version: "encryptionkey@0"
I20260808 06:00:03.190640 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "c5c79d04ea0e4ad68fdbc6b52fb1d0d0"
format_stamp: "Formatted at 2026-08-08 06:00:03 on dist-test-slave-zkpd"
server_key: "ca38b108f18be6809292219cea954aa9"
server_key_iv: "6c2ecc7904fe73d89757c61cb556ac68"
server_key_version: "encryptionkey@0"
I20260808 06:00:03.194278 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:03.196733 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:03.198001 17836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:03.198130 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "c5c79d04ea0e4ad68fdbc6b52fb1d0d0"
format_stamp: "Formatted at 2026-08-08 06:00:03 on dist-test-slave-zkpd"
server_key: "ca38b108f18be6809292219cea954aa9"
server_key_iv: "6c2ecc7904fe73d89757c61cb556ac68"
server_key_version: "encryptionkey@0"
I20260808 06:00:03.198241 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:03.208400 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:03.211398 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:03.211598 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:03.212271 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:03.213099 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:03.213168 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:03.213239 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:03.213282 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:03.222412 17836 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38483
I20260808 06:00:03.222440 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38483 every 8 connection(s)
I20260808 06:00:03.223276 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:03.228468 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17836
I20260808 06:00:03.228603 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:03.228915 14229 external_mini_cluster.cc:1428] Setting key e0129b22dba1ccaab8b80bb6c0bf6083
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168803, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:03.238396 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:03.225151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:48925 (local address 127.13.229.126:35089)
0808 06:00:03.225301 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:03.225305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:03.226121 (+ 816us) server_negotiation.cc:407] Connection header received
0808 06:00:03.227471 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:03.227473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:03.227521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:03.227595 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:03.228969 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.229660 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.231465 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.231678 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.231766 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:03.234715 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:03.234718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:03.234720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:03.234748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:03.236452 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.236970 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.236973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.236974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.237056 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.237379 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.237381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.237382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.237573 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:03.237657 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:03.238148 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 06:00:03.238257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:00:03.239316 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:03.225520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35089 (local address 127.13.229.67:48925)
0808 06:00:03.225971 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:03.226011 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:03.227281 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:03.227787 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:03.227794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:03.228218 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:03.228818 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:03.228834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.229879 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.229885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:03.231341 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:03.231352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.231845 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.231848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:03.232145 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:03.232855 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:03.232873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:03.234551 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:03.236580 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:03.236585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:03.236598 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:03.236873 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:03.237145 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:03.237147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:03.237149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:03.237283 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:03.237665 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:03.237670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:03.237920 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 06:00:03.238080 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 06:00:03.238341 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
I20260808 06:00:03.240752 17970 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35089
I20260808 06:00:03.241010 17970 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:03.241534 17970 heartbeater.cc:511] Master 127.13.229.126:35089 requested a full tablet report, sending...
I20260808 06:00:03.242791 17489 ts_manager.cc:194] Registered new tserver with Master: c5c79d04ea0e4ad68fdbc6b52fb1d0d0 (127.13.229.67:38483)
I20260808 06:00:03.243453 17489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:48925
I20260808 06:00:03.255136 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168803, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168803, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:03.283190 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:03.274300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.13.229.126:35089)
0808 06:00:03.274447 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:03.274451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:03.274608 (+ 157us) server_negotiation.cc:407] Connection header received
0808 06:00:03.274725 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:03.274728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:03.274771 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:03.274850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:03.275545 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.276126 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.277178 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:03.277402 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:03.277485 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:03.280088 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:03.280091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:03.280092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:03.280116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:03.281567 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.281995 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.281997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.281999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.282074 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:03.282359 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:03.282371 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:03.282372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:03.282584 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:03.282686 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:03.282964 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 06:00:03.283063 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 06:00:03.283728 17489 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43852
I20260808 06:00:03.284719 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17555
I20260808 06:00:03.292563 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17698
I20260808 06:00:03.299669 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17836
I20260808 06:00:03.305577 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17456
2026-08-08T06:00:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2201 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17983](info): set up 2 sockets
Aug 08 06:00:03 dist-test-slave-zkpd krb5kdc[17983](info): commencing operation
krb5kdc: starting...
W20260808 06:00:04.377378 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:04 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:04.506884 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43865
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:43865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:04.622138 17997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:04.622452 17997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:04.622536 17997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:04.626637 17997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:04.626739 17997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:04.626778 17997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:04.626818 17997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:04.626848 17997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:04.632313 17997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:43865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:04.633533 17997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:04.634653 17997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:04.643390 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:04.643481 18002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:04.643505 18003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:04.643661 17997 server_base.cc:1034] running on GCE node
I20260808 06:00:04.644409 17997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:04.645529 17997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:04.646735 17997 hybrid_clock.cc:648] HybridClock initialized: now 1786168804646703 us; error 49 us; skew 500 ppm
Aug 08 06:00:04 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:04.649631 17997 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:04.650533 17997 webserver.cc:460] Webserver started at http://127.13.229.126:40621/ using document root <none> and password file <none>
I20260808 06:00:04.651110 17997 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:04.651204 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:04.651451 17997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:04.653682 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "07b356bdcdf74512a16c32b23d5d5f06"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "436c7406244b7ab2647e087f952a54df"
server_key_iv: "cdbd1e16ee9ea82f5aa5c3f338019414"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.654278 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "07b356bdcdf74512a16c32b23d5d5f06"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "436c7406244b7ab2647e087f952a54df"
server_key_iv: "cdbd1e16ee9ea82f5aa5c3f338019414"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.658166 17997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 06:00:04.660672 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:04.661919 17997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:00:04.662074 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "07b356bdcdf74512a16c32b23d5d5f06"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "436c7406244b7ab2647e087f952a54df"
server_key_iv: "cdbd1e16ee9ea82f5aa5c3f338019414"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.662195 17997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:04.682487 17997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:04.686015 17997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:04.686239 17997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:04.693351 17997 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:43865
I20260808 06:00:04.693435 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:43865 every 8 connection(s)
I20260808 06:00:04.694427 17997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:04.697374 18069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:04.703068 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06: Bootstrap starting.
I20260808 06:00:04.704212 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 17997
I20260808 06:00:04.704355 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:04.704632 14229 external_mini_cluster.cc:1428] Setting key 69465e2c0e6150984e542255bf007ef5
I20260808 06:00:04.705984 18069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:04.706950 18069 log.cc:824] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:04.709275 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06: No bootstrap required, opened a new log
I20260808 06:00:04.713351 18069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } }
I20260808 06:00:04.713660 18069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:04.713721 18069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b356bdcdf74512a16c32b23d5d5f06, State: Initialized, Role: FOLLOWER
Aug 08 06:00:04 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:04.714275 18069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } }
I20260808 06:00:04.714464 18069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:04.714543 18069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:04.714651 18069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:04.715618 18069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } }
I20260808 06:00:04.716009 18069 leader_election.cc:302] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07b356bdcdf74512a16c32b23d5d5f06; no voters:
I20260808 06:00:04.716419 18069 leader_election.cc:288] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:04.716503 18074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:04.716727 18074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [term 1 LEADER]: Becoming Leader. State: Replica: 07b356bdcdf74512a16c32b23d5d5f06, State: Running, Role: LEADER
I20260808 06:00:04.717113 18074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } }
I20260808 06:00:04.717859 18069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:04.718961 18075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07b356bdcdf74512a16c32b23d5d5f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } } }
I20260808 06:00:04.719089 18075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:04.719684 18076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b356bdcdf74512a16c32b23d5d5f06. Latest consensus state: current_term: 1 leader_uuid: "07b356bdcdf74512a16c32b23d5d5f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b356bdcdf74512a16c32b23d5d5f06" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43865 } } }
I20260808 06:00:04.719911 18076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:04.720723 18083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:04.721098 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:04.705812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.13.229.126:43865)
0808 06:00:04.706370 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:04.706390 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:04.706435 (+ 45us) server_negotiation.cc:407] Connection header received
0808 06:00:04.707308 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:04.707338 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:04.707732 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:04.708130 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:04.708792 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.710223 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.711076 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.711798 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.712002 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:04.714799 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:04.714805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:04.714825 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:04.714859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:04.717053 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:04.717589 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:04.717594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:04.717599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:04.717709 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:04.718066 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:04.718070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:04.718071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:04.718642 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:04.719497 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:04.719659 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 06:00:04.720017 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":182,"threads_started":1}
I20260808 06:00:04.724144 18083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:04.730213 18083 catalog_manager.cc:1347] Generated new cluster ID: 3f355db45e1647c0a1c823b13cb67657
I20260808 06:00:04.730290 18083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:04.746351 18083 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:04.747675 18083 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:04.755564 18083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07b356bdcdf74512a16c32b23d5d5f06: Generated new TSK 0
I20260808 06:00:04.756441 18083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:04.787590 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:04.903219 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:04.903513 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:04.903599 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:04.907529 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:04.907581 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:04.907637 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:04.912690 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:04.913766 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:04.914772 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:04.924397 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:04.924397 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:04.924453 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:04.924724 18095 server_base.cc:1034] running on GCE node
I20260808 06:00:04.925132 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:04.925769 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:04.926954 18095 hybrid_clock.cc:648] HybridClock initialized: now 1786168804926933 us; error 30 us; skew 500 ppm
Aug 08 06:00:04 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:04.929912 18095 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:04.930904 18095 webserver.cc:460] Webserver started at http://127.13.229.65:44059/ using document root <none> and password file <none>
I20260808 06:00:04.931471 18095 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:04.931562 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:04.931794 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:04.933895 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "bf6d1d4cb9264e7b9209337848b64fcd"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "1095e347d6c5f6039d2451dbb20a2a68"
server_key_iv: "a8adce77e01f435de181a22a850aabda"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.934478 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "bf6d1d4cb9264e7b9209337848b64fcd"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "1095e347d6c5f6039d2451dbb20a2a68"
server_key_iv: "a8adce77e01f435de181a22a850aabda"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.938217 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:04.940588 18110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:04.941857 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:04.941995 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "bf6d1d4cb9264e7b9209337848b64fcd"
format_stamp: "Formatted at 2026-08-08 06:00:04 on dist-test-slave-zkpd"
server_key: "1095e347d6c5f6039d2451dbb20a2a68"
server_key_iv: "a8adce77e01f435de181a22a850aabda"
server_key_version: "encryptionkey@0"
I20260808 06:00:04.942106 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:04.953900 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:04.957314 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:04.957525 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:04.958163 18095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:04.958997 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:04.959066 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:04.959138 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:04.959189 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:04.968974 18095 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:44707
I20260808 06:00:04.968997 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:44707 every 8 connection(s)
I20260808 06:00:04.969913 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:04.974869 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18095
I20260808 06:00:04.975015 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:04.975378 14229 external_mini_cluster.cc:1428] Setting key 3abfc96dfcefdc29b70e7bf198200042
Aug 08 06:00:04 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168804, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:04.985559 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:04.971731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:58311 (local address 127.13.229.126:43865)
0808 06:00:04.971885 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:04.971922 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:04.972873 (+ 951us) server_negotiation.cc:407] Connection header received
0808 06:00:04.974317 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:04.974320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:04.974381 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:04.974491 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:04.976115 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.976655 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.978344 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.978583 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.978663 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:04.981577 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:04.981580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:04.981582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:04.981606 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:04.983182 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:04.983868 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:04.983871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:04.983872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:04.983999 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:04.984415 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:04.984417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:04.984418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:04.984594 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:04.984667 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:04.985333 (+ 666us) server_negotiation.cc:299] Negotiation successful
0808 06:00:04.985435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:00:04.986801 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:04.972223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43865 (local address 127.13.229.65:58311)
0808 06:00:04.972694 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:04.972737 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:04.974089 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:04.974693 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:04.974704 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:04.975248 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:04.975955 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:04.975984 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.976794 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.976798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:04.978198 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:04.978213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:04.978738 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:04.978746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:04.978993 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:04.979656 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:04.979672 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:04.981397 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:04.983311 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:04.983320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:04.983341 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:04.983752 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:04.984095 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:04.984101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:04.984106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:04.984310 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:04.984700 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:04.984708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:04.985062 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 06:00:04.985289 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 06:00:04.985652 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260808 06:00:04.988176 18226 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43865
I20260808 06:00:04.988435 18226 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:04.988937 18226 heartbeater.cc:511] Master 127.13.229.126:43865 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:04.990665 18031 ts_manager.cc:194] Registered new tserver with Master: bf6d1d4cb9264e7b9209337848b64fcd (127.13.229.65:44707)
I20260808 06:00:04.992048 18031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:58311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:05.005093 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:05.123425 18231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:05.123694 18231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:05.123739 18231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:05.127749 18231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:05.127825 18231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:05.127883 18231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:05.134083 18231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:05.135147 18231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:05.136214 18231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:05.145792 18236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:05.145782 18237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:05.145792 18239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:05.146099 18231 server_base.cc:1034] running on GCE node
I20260808 06:00:05.146575 18231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:05.147197 18231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:05.148352 18231 hybrid_clock.cc:648] HybridClock initialized: now 1786168805148337 us; error 25 us; skew 500 ppm
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:05.151093 18231 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:05.151981 18231 webserver.cc:460] Webserver started at http://127.13.229.66:43737/ using document root <none> and password file <none>
I20260808 06:00:05.152536 18231 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:05.152624 18231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:05.152856 18231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:05.154894 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "968e6baec00d4c169046fe3fce3740d8"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "206f3c94ff9ce778959129785315796c"
server_key_iv: "ddb866f1512e2050c1ceb2ad2ccf6f54"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.155478 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "968e6baec00d4c169046fe3fce3740d8"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "206f3c94ff9ce778959129785315796c"
server_key_iv: "ddb866f1512e2050c1ceb2ad2ccf6f54"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.159222 18231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:00:05.161801 18246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.162988 18231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:00:05.163130 18231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "968e6baec00d4c169046fe3fce3740d8"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "206f3c94ff9ce778959129785315796c"
server_key_iv: "ddb866f1512e2050c1ceb2ad2ccf6f54"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.163244 18231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:05.177145 18231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:05.180549 18231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:05.180761 18231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:05.181358 18231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:05.182324 18231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:05.182394 18231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.182463 18231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:05.182510 18231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.191141 18231 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:37465
I20260808 06:00:05.191213 18363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:37465 every 8 connection(s)
I20260808 06:00:05.192078 18231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:05.202020 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18231
I20260808 06:00:05.202198 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:05.202549 14229 external_mini_cluster.cc:1428] Setting key 0a4516bed5b6cd52bfbb0352793f5346
I20260808 06:00:05.205855 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.193933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:60679 (local address 127.13.229.126:43865)
0808 06:00:05.194087 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:05.194091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:05.194995 (+ 904us) server_negotiation.cc:407] Connection header received
0808 06:00:05.196402 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:05.196405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:05.196451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:05.196527 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:05.197663 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.198173 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.199250 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.199461 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.199539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.202135 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:05.202142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:05.202148 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:05.202174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:05.203639 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.204258 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.204260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.204262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.204339 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.204740 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.204742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.204743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.204920 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:05.204995 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:05.205637 (+ 642us) server_negotiation.cc:299] Negotiation successful
0808 06:00:05.205739 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:00:05.206899 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.194367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43865 (local address 127.13.229.66:60679)
0808 06:00:05.194816 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:05.194858 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:05.196210 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:05.196694 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:05.196700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:05.197068 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:05.197537 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:05.197549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.198279 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.198281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:05.199146 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:05.199154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.199589 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.199592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:05.199736 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.200413 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:05.200429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:05.201935 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:05.203752 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:05.203758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:05.203774 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:05.204160 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:05.204443 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:05.204446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:05.204448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:05.204641 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:05.205025 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:05.205035 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:05.205380 (+ 345us) client_negotiation.cc:769] Sending connection context
0808 06:00:05.205587 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 06:00:05.205948 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260808 06:00:05.208106 18364 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43865
I20260808 06:00:05.208370 18364 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:05.208948 18364 heartbeater.cc:511] Master 127.13.229.126:43865 requested a full tablet report, sending...
I20260808 06:00:05.210144 18031 ts_manager.cc:194] Registered new tserver with Master: 968e6baec00d4c169046fe3fce3740d8 (127.13.229.66:37465)
I20260808 06:00:05.210713 18031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:60679
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:05.233032 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:05.350658 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:05.350912 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:05.350982 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:05.355182 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:05.355242 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:05.355335 18369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:05.360639 18369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:05.361712 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:05.362766 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:05.372572 18375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:05.372574 18377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:05.372591 18374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:05.372797 18369 server_base.cc:1034] running on GCE node
I20260808 06:00:05.373385 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:05.374006 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:05.375200 18369 hybrid_clock.cc:648] HybridClock initialized: now 1786168805375172 us; error 46 us; skew 500 ppm
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:05.378119 18369 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:05.378968 18369 webserver.cc:460] Webserver started at http://127.13.229.67:41603/ using document root <none> and password file <none>
I20260808 06:00:05.379529 18369 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:05.379621 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:05.379858 18369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:05.382019 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "14af36bc95894f83a26be3a61cfae438"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "b394bf217e7cebf196fc1a6821e46513"
server_key_iv: "d17fa7a1416ff4e8cd59f6c6f9350013"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.382665 18369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "14af36bc95894f83a26be3a61cfae438"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "b394bf217e7cebf196fc1a6821e46513"
server_key_iv: "d17fa7a1416ff4e8cd59f6c6f9350013"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.386425 18369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:00:05.388820 18384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.390031 18369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:05.390167 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "14af36bc95894f83a26be3a61cfae438"
format_stamp: "Formatted at 2026-08-08 06:00:05 on dist-test-slave-zkpd"
server_key: "b394bf217e7cebf196fc1a6821e46513"
server_key_iv: "d17fa7a1416ff4e8cd59f6c6f9350013"
server_key_version: "encryptionkey@0"
I20260808 06:00:05.390288 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:05.406613 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:05.409765 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:05.409991 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:05.410620 18369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:05.411491 18369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:05.411563 18369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.411638 18369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:05.411684 18369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:05.421474 18369 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:46837
I20260808 06:00:05.421504 18502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:46837 every 8 connection(s)
I20260808 06:00:05.422415 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:05.430419 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18369
I20260808 06:00:05.430528 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:05.430940 14229 external_mini_cluster.cc:1428] Setting key 99be950b5456c1dbbcd630420bce4f39
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:05.436223 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.424282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:41537 (local address 127.13.229.126:43865)
0808 06:00:05.424438 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:05.424442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:05.425297 (+ 855us) server_negotiation.cc:407] Connection header received
0808 06:00:05.426711 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:05.426714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:05.426775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:05.426853 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:05.428078 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.428658 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.429833 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.430066 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.430188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.432869 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:05.432873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:05.432875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:05.432908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:05.434291 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.434785 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.434789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.434790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.434882 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.435187 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.435189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.435190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.435381 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:05.435458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:05.436002 (+ 544us) server_negotiation.cc:299] Negotiation successful
0808 06:00:05.436106 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 06:00:05.437093 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.424675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43865 (local address 127.13.229.67:41537)
0808 06:00:05.425140 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:05.425182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:05.426487 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:05.427041 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:05.427049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:05.427401 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:05.427917 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:05.427933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.428794 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.428804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:05.429677 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:05.429708 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.430241 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.430245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:05.430416 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.431198 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:05.431214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:05.432712 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:05.434405 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:05.434410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:05.434423 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:05.434675 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:05.434971 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:05.434973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:05.434975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:05.435105 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:05.435469 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:05.435474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:05.435727 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 06:00:05.435935 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 06:00:05.436203 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260808 06:00:05.438273 18503 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43865
I20260808 06:00:05.438519 18503 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:05.439113 18503 heartbeater.cc:511] Master 127.13.229.126:43865 requested a full tablet report, sending...
I20260808 06:00:05.440320 18031 ts_manager.cc:194] Registered new tserver with Master: 14af36bc95894f83a26be3a61cfae438 (127.13.229.67:46837)
I20260808 06:00:05.440995 18031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:41537
I20260808 06:00:05.445152 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 06:00:05.475622 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.465493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.13.229.65:44707)
0808 06:00:05.465850 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:05.465866 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:05.465892 (+ 26us) server_negotiation.cc:407] Connection header received
0808 06:00:05.465943 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:05.465948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:05.466145 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:05.466270 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:05.467187 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.468257 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.469126 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.469515 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.469603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.472074 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:05.472084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:05.472099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:05.472126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:05.473931 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.474312 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.474318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.474319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.474400 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.474626 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.474628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.474629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.474912 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:05.475097 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:05.475240 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 06:00:05.475414 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[17983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:05.487319 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:05.478761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.13.229.126:43865)
0808 06:00:05.478900 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:05.478904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:05.479066 (+ 162us) server_negotiation.cc:407] Connection header received
0808 06:00:05.479219 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:05.479224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:05.479274 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:05.479371 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:05.480013 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.480576 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.481387 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:05.481610 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:05.481720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:05.484253 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:05.484257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:05.484259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:05.484285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:05.485784 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.486143 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.486146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.486147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.486227 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:05.486464 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:05.486466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:05.486467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:05.486690 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:05.486797 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:05.486989 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:00:05.487123 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 06:00:05.487749 18031 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56974
I20260808 06:00:05.488481 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18095
I20260808 06:00:05.496330 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18231
I20260808 06:00:05.504081 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18369
I20260808 06:00:05.510365 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 17997
2026-08-08T06:00:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2203 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[18515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[18515](info): set up 2 sockets
Aug 08 06:00:05 dist-test-slave-zkpd krb5kdc[18515](info): commencing operation
krb5kdc: starting...
W20260808 06:00:06.561671 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:06 dist-test-slave-zkpd krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:06.690801 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41265
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:41265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:06.804847 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:06.805166 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:06.805251 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:06.809327 18529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:06.809412 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:06.809453 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:06.809511 18529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:06.809551 18529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:06.814832 18529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:41265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:06.816052 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:06.817099 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:06.825515 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:06.825591 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:06.825652 18529 server_base.cc:1034] running on GCE node
W20260808 06:00:06.825515 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:06.826396 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:06.827509 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:06.828670 18529 hybrid_clock.cc:648] HybridClock initialized: now 1786168806828655 us; error 31 us; skew 500 ppm
Aug 08 06:00:06 dist-test-slave-zkpd krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168806, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:06.835685 18529 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:06.836521 18529 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:06.836592 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:06.836764 18529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:06.838950 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "b368d864f8704644996bd77b671c7e63"
format_stamp: "Formatted at 2026-08-08 06:00:06 on dist-test-slave-zkpd"
server_key: "be3805d9179ba8712b07b73491fed495"
server_key_iv: "6593700d7300a4ed90b1bc0023a176ae"
server_key_version: "encryptionkey@0"
I20260808 06:00:06.839486 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "b368d864f8704644996bd77b671c7e63"
format_stamp: "Formatted at 2026-08-08 06:00:06 on dist-test-slave-zkpd"
server_key: "be3805d9179ba8712b07b73491fed495"
server_key_iv: "6593700d7300a4ed90b1bc0023a176ae"
server_key_version: "encryptionkey@0"
I20260808 06:00:06.843186 18529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:00:06.845603 18543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:06.846824 18529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:06.846958 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "b368d864f8704644996bd77b671c7e63"
format_stamp: "Formatted at 2026-08-08 06:00:06 on dist-test-slave-zkpd"
server_key: "be3805d9179ba8712b07b73491fed495"
server_key_iv: "6593700d7300a4ed90b1bc0023a176ae"
server_key_version: "encryptionkey@0"
I20260808 06:00:06.847098 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:06.873974 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:06.877576 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:06.877739 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:06.884641 18529 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:41265
I20260808 06:00:06.884949 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:41265 every 8 connection(s)
I20260808 06:00:06.885334 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:06.887830 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18529
I20260808 06:00:06.887998 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:06.888206 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:06.888327 14229 external_mini_cluster.cc:1428] Setting key 94122ff33db1825b012d9d1ebbd4febf
I20260808 06:00:06.895577 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: Bootstrap starting.
Aug 08 06:00:06 dist-test-slave-zkpd krb5kdc[18515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:06.897926 18599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:06.898823 18599 log.cc:824] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:06.901165 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: No bootstrap required, opened a new log
I20260808 06:00:06.903239 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:06.889746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.13.229.126:41265)
0808 06:00:06.890228 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:06.890253 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:06.890300 (+ 47us) server_negotiation.cc:407] Connection header received
0808 06:00:06.891014 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:06.891038 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:06.891472 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:06.891845 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:06.892602 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:06.894078 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:06.894981 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:06.895665 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:06.895821 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:06.898447 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:06.898453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:06.898470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:06.898500 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:06.900478 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:06.900964 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:06.900969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:06.900973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:06.901081 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:06.901389 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:06.901392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:06.901393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:06.901781 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:06.901928 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:06.902227 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 06:00:06.902529 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":119,"threads_started":1}
I20260808 06:00:06.905364 18599 raft_consensus.cc:348] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } }
I20260808 06:00:06.905622 18599 raft_consensus.cc:374] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:06.905687 18599 raft_consensus.cc:729] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b368d864f8704644996bd77b671c7e63, State: Initialized, Role: FOLLOWER
I20260808 06:00:06.906103 18599 consensus_queue.cc:260] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } }
I20260808 06:00:06.906243 18599 raft_consensus.cc:388] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:06.906306 18599 raft_consensus.cc:482] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:06.906427 18599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:06.907444 18599 raft_consensus.cc:504] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } }
I20260808 06:00:06.907788 18599 leader_election.cc:302] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b368d864f8704644996bd77b671c7e63; no voters:
I20260808 06:00:06.908131 18599 leader_election.cc:288] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:06.908277 18604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:06.908497 18604 raft_consensus.cc:686] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [term 1 LEADER]: Becoming Leader. State: Replica: b368d864f8704644996bd77b671c7e63, State: Running, Role: LEADER
I20260808 06:00:06.908833 18604 consensus_queue.cc:237] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } }
I20260808 06:00:06.909343 18599 sys_catalog.cc:564] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:06.910491 18605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b368d864f8704644996bd77b671c7e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } } }
I20260808 06:00:06.910630 18605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:06.910845 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b368d864f8704644996bd77b671c7e63. Latest consensus state: current_term: 1 leader_uuid: "b368d864f8704644996bd77b671c7e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b368d864f8704644996bd77b671c7e63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41265 } } }
I20260808 06:00:06.911017 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:06.911829 18613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:06.915109 18613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:06.921026 18613 catalog_manager.cc:1347] Generated new cluster ID: 537281102b83421bb12331645ab7ce6d
I20260808 06:00:06.921093 18613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:06.939159 18613 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:06.940332 18613 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:06.959913 18613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: Generated new TSK 0
I20260808 06:00:06.960760 18613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:06.994915 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:07.113427 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:07.113708 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:07.113818 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:07.117859 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:07.117918 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:07.118022 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:07.123456 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:07.124574 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:07.125649 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:07.135429 18627 server_base.cc:1034] running on GCE node
W20260808 06:00:07.135381 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.135344 18632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.135350 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:07.136255 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:07.136909 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:07.138104 18627 hybrid_clock.cc:648] HybridClock initialized: now 1786168807138089 us; error 26 us; skew 500 ppm
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:07.141182 18627 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:07.141922 18627 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:07.142021 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:07.142259 18627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:07.144402 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "263186d2ae004bd290de9e9e148e88f4"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "6b8cba0b281df293f7d6aa6d116505b8"
server_key_iv: "6c91157d32b5be415dd73b2ec47d6694"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.144979 18627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "263186d2ae004bd290de9e9e148e88f4"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "6b8cba0b281df293f7d6aa6d116505b8"
server_key_iv: "6c91157d32b5be415dd73b2ec47d6694"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.148882 18627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:00:07.151492 18641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.152801 18627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:07.152941 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "263186d2ae004bd290de9e9e148e88f4"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "6b8cba0b281df293f7d6aa6d116505b8"
server_key_iv: "6c91157d32b5be415dd73b2ec47d6694"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.153054 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:07.176419 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:07.180011 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:07.180219 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:07.180852 18627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:07.181735 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:07.181807 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.181875 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:07.181926 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.191061 18627 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:36711
I20260808 06:00:07.191080 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:36711 every 8 connection(s)
I20260808 06:00:07.191674 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:07.192538 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18627
I20260808 06:00:07.192693 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:07.193025 14229 external_mini_cluster.cc:1428] Setting key 41a690210237d8b9ddfc80473b4f2f92
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:07.207294 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.194771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48369 (local address 127.13.229.126:41265)
0808 06:00:07.194927 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.194932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.195846 (+ 914us) server_negotiation.cc:407] Connection header received
0808 06:00:07.197163 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.197167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.197220 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.197305 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:07.198478 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.199016 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.200239 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.200496 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.200620 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.203525 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:07.203528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:07.203531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:07.203555 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:07.205010 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.205505 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.205508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.205509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.205592 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.205971 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.205976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.205977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.206216 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:07.206297 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.206871 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.207030 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:00:07.208034 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.195196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41265 (local address 127.13.229.65:48369)
0808 06:00:07.195683 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.195736 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.196950 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.197496 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.197503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.197854 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:07.198331 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.198345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.199143 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.199146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.200117 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.200127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.200662 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.200667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.200832 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.201478 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:07.201493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:07.203286 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:07.205118 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.205123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.205135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.205395 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.205704 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.205706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.205708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.205862 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.206326 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:07.206336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:07.206622 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.206795 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.207050 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":143,"threads_started":1}
I20260808 06:00:07.209487 18761 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41265
I20260808 06:00:07.209748 18761 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:07.210366 18761 heartbeater.cc:511] Master 127.13.229.126:41265 requested a full tablet report, sending...
I20260808 06:00:07.212049 18561 ts_manager.cc:194] Registered new tserver with Master: 263186d2ae004bd290de9e9e148e88f4 (127.13.229.65:36711)
I20260808 06:00:07.213171 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:48369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:07.223100 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:07.339815 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:07.340137 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:07.340185 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:07.344144 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:07.344195 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:07.344254 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:07.349400 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:07.350487 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:07.351572 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:07.361276 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.361335 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.361276 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:07.361647 18766 server_base.cc:1034] running on GCE node
I20260808 06:00:07.362072 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:07.362748 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:07.363963 18766 hybrid_clock.cc:648] HybridClock initialized: now 1786168807363941 us; error 29 us; skew 500 ppm
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:07.367058 18766 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:07.367782 18766 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:07.367846 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:07.368110 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:07.370237 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "3b117e651149409aaecc3468b4295c1c"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "da339cf26ee3d48cb9cfe3af859d955d"
server_key_iv: "55c00d2236e380a4dc033c5e9306bdbf"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.370811 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "3b117e651149409aaecc3468b4295c1c"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "da339cf26ee3d48cb9cfe3af859d955d"
server_key_iv: "55c00d2236e380a4dc033c5e9306bdbf"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.374564 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:00:07.377012 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.378348 18766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:07.378485 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "3b117e651149409aaecc3468b4295c1c"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "da339cf26ee3d48cb9cfe3af859d955d"
server_key_iv: "55c00d2236e380a4dc033c5e9306bdbf"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.378595 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:07.394358 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:07.397388 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:07.397599 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:07.398207 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:07.399050 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:07.399122 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.399199 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:07.399242 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.408567 18766 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:45531
I20260808 06:00:07.408586 18897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:45531 every 8 connection(s)
I20260808 06:00:07.409229 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:07.410667 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18766
I20260808 06:00:07.410796 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:07.411181 14229 external_mini_cluster.cc:1428] Setting key f019b6d844c9fea693e5c985afb7bf77
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:00:07.425642 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.411365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:46987 (local address 127.13.229.126:41265)
0808 06:00:07.411532 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.411538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.412694 (+ 1156us) server_negotiation.cc:407] Connection header received
0808 06:00:07.414600 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.414603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.414655 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.414731 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:07.416071 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.416757 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.418615 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.418841 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.418926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.421866 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:07.421870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:07.421873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:07.421907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:07.423465 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.424024 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.424027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.424029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.424116 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.424474 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.424477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.424479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.424675 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:07.424785 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.425399 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.425517 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:00:07.426885 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.411927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41265 (local address 127.13.229.66:46987)
0808 06:00:07.412519 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.412587 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.414355 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.414951 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.414963 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.415381 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:07.415865 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.415880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.416906 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.416913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.418483 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.418498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.419014 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.419021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.419279 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.420008 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:07.420036 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:07.421687 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:07.423590 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.423596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.423612 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.423927 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.424210 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.424214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.424216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.424366 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.424791 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:07.424801 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:07.425101 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.425311 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.425703 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":136,"threads_started":1}
I20260808 06:00:07.428185 18898 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41265
I20260808 06:00:07.428462 18898 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:07.428949 18898 heartbeater.cc:511] Master 127.13.229.126:41265 requested a full tablet report, sending...
I20260808 06:00:07.430061 18561 ts_manager.cc:194] Registered new tserver with Master: 3b117e651149409aaecc3468b4295c1c (127.13.229.66:45531)
I20260808 06:00:07.430647 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:46987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:07.443049 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:07.562719 18903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:07.563015 18903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:07.563093 18903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:07.567109 18903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:07.567199 18903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:07.567291 18903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:07.572646 18903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:07.573765 18903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:07.574802 18903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:07.584769 18909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.584808 18908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:07.584774 18911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:07.585234 18903 server_base.cc:1034] running on GCE node
I20260808 06:00:07.585660 18903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:07.586295 18903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:07.587491 18903 hybrid_clock.cc:648] HybridClock initialized: now 1786168807587472 us; error 29 us; skew 500 ppm
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:07.590342 18903 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:07.591116 18903 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:07.591213 18903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:07.591455 18903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:07.593619 18903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "e4940ff1600146a1bf4049daf234c242"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "73a5b7b98cd0b615d2571b4ffc551009"
server_key_iv: "721650c76daeab693da1f691861a8415"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.594195 18903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "e4940ff1600146a1bf4049daf234c242"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "73a5b7b98cd0b615d2571b4ffc551009"
server_key_iv: "721650c76daeab693da1f691861a8415"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.598234 18903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 06:00:07.600719 18917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.601881 18903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:07.602015 18903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "e4940ff1600146a1bf4049daf234c242"
format_stamp: "Formatted at 2026-08-08 06:00:07 on dist-test-slave-zkpd"
server_key: "73a5b7b98cd0b615d2571b4ffc551009"
server_key_iv: "721650c76daeab693da1f691861a8415"
server_key_version: "encryptionkey@0"
I20260808 06:00:07.602128 18903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:07.616605 18903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:07.619756 18903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:07.620054 18903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:07.620694 18903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:07.621565 18903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:07.621636 18903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.621709 18903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:07.621760 18903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:07.631207 18903 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:39735
I20260808 06:00:07.631224 19033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:39735 every 8 connection(s)
I20260808 06:00:07.631820 18903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:07.640532 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 18903
I20260808 06:00:07.640681 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:07.640988 14229 external_mini_cluster.cc:1428] Setting key 598f9d93a6fa9c3ff87d3165d67f3a23
Aug 08 06:00:07 dist-test-slave-zkpd krb5kdc[18515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168807, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:07.646190 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.633843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:48145 (local address 127.13.229.126:41265)
0808 06:00:07.633998 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.634002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.634820 (+ 818us) server_negotiation.cc:407] Connection header received
0808 06:00:07.636170 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.636172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.636221 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.636300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:07.637547 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.638105 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.639322 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.639533 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.639611 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.642632 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:07.642636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:07.642638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:07.642665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:07.644236 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.644715 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.644718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.644719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.644820 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.645152 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.645155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.645156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.645347 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:07.645440 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.645960 (+ 520us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.646069 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:00:07.647032 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.634193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41265 (local address 127.13.229.67:48145)
0808 06:00:07.634665 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.634710 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.635968 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.636486 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.636493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.636898 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:07.637409 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.637424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.638229 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.638232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.639211 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.639221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.639669 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.639672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.639852 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.640627 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:07.640677 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:07.642446 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:07.644350 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.644355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.644366 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.644624 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.644901 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:07.644903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:07.644904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:07.645035 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:07.645478 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:07.645484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:07.645709 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.645895 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.646145 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260808 06:00:07.648325 19034 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41265
I20260808 06:00:07.648567 19034 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:07.649037 19034 heartbeater.cc:511] Master 127.13.229.126:41265 requested a full tablet report, sending...
I20260808 06:00:07.650200 18561 ts_manager.cc:194] Registered new tserver with Master: e4940ff1600146a1bf4049daf234c242 (127.13.229.67:39735)
I20260808 06:00:07.650733 18561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:48145
I20260808 06:00:07.654806 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:00:07.664191 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.656694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.13.229.126:41265)
0808 06:00:07.656811 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.656815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.657054 (+ 239us) server_negotiation.cc:407] Connection header received
0808 06:00:07.657202 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.657206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.657249 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.657343 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:07.658120 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.658643 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.659478 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.659714 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.659821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.661249 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:07.661252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:07.661255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:07.661280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:07.662457 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.662890 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.662894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.662895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.662982 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:07.663257 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:07.663262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:07.663263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:07.663485 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:07.663630 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.663879 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.664025 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:00:07.667220 18561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:07.670316 18561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:07.685405 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.680182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.13.229.67:39735)
0808 06:00:07.680744 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.680754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.680775 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:00:07.680847 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.680851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.681020 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.681170 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:07.681369 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682546 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.684070 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.684674 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.684816 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.684972 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.685113 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.685221 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":165,"threads_started":1}
I20260808 06:00:07.685453 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.678945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:45531 (local address 127.0.0.1:51970)
0808 06:00:07.680102 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.680115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.680192 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.680787 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.680794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.680819 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:07.680967 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.680974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682273 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.682276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.683844 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.683856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.684963 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.684967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.685060 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.685083 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.685134 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.685292 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":48,"thread_start_us":98,"threads_started":1}
I20260808 06:00:07.685559 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.679261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.13.229.65:36711)
0808 06:00:07.679960 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.679967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.680297 (+ 330us) server_negotiation.cc:407] Connection header received
0808 06:00:07.680386 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.680392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.680621 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.680775 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:07.681244 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682424 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.683859 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.684724 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.684834 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.685005 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.685310 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.685400 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":90,"threads_started":1}
I20260808 06:00:07.685943 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.678945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:36711 (local address 127.0.0.1:37236)
0808 06:00:07.679947 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.680001 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.680145 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.680969 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.680972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.680990 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:07.681115 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.681125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682580 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.682583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.683729 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.683740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.685060 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.685063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.685155 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.685172 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.685232 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.685817 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":611,"thread_start_us":110,"threads_started":1}
I20260808 06:00:07.686110 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.679771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:39735 (local address 127.0.0.1:57834)
0808 06:00:07.680392 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.680405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.680477 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.681142 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.681144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.681168 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:07.681249 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.681255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682682 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.682686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.683973 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.683982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.684810 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.684814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.684921 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.684958 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.685855 (+ 897us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.685938 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260808 06:00:07.686295 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.679773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.13.229.66:45531)
0808 06:00:07.680267 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.680281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.680306 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:00:07.680394 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.680399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.680625 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.680806 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:07.681115 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.682077 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.683989 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.684828 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.685833 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.685981 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.686038 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.686131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":134,"threads_started":1}
I20260808 06:00:07.689016 18967 tablet_service.cc:1467] Processing CreateTablet for tablet 6d66efba87f54d6cac63157231fd1c50 (DEFAULT_TABLE table=test-table [id=64e0d5a6d85040448085bf262c428b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:07.689059 18690 tablet_service.cc:1467] Processing CreateTablet for tablet 6d66efba87f54d6cac63157231fd1c50 (DEFAULT_TABLE table=test-table [id=64e0d5a6d85040448085bf262c428b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:07.689059 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 6d66efba87f54d6cac63157231fd1c50 (DEFAULT_TABLE table=test-table [id=64e0d5a6d85040448085bf262c428b08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:07.690163 18967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d66efba87f54d6cac63157231fd1c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:07.690165 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d66efba87f54d6cac63157231fd1c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:07.690165 18690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d66efba87f54d6cac63157231fd1c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:07.696596 19052 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Bootstrap starting.
I20260808 06:00:07.698347 19053 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Bootstrap starting.
I20260808 06:00:07.698951 19052 tablet_bootstrap.cc:654] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:07.700330 19052 log.cc:824] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:07.700544 19053 tablet_bootstrap.cc:654] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:07.701058 19054 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: Bootstrap starting.
I20260808 06:00:07.701826 19053 log.cc:824] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:07.702665 19052 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: No bootstrap required, opened a new log
I20260808 06:00:07.702880 19052 ts_tablet_manager.cc:1397] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:07.703065 19054 tablet_bootstrap.cc:654] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:07.703939 19053 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: No bootstrap required, opened a new log
I20260808 06:00:07.704123 19053 ts_tablet_manager.cc:1397] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 06:00:07.704095 19054 log.cc:824] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:07.705914 19054 tablet_bootstrap.cc:492] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: No bootstrap required, opened a new log
I20260808 06:00:07.706104 19054 ts_tablet_manager.cc:1397] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:00:07.707195 19052 raft_consensus.cc:348] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.707588 19052 raft_consensus.cc:374] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:07.707681 19052 raft_consensus.cc:729] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4940ff1600146a1bf4049daf234c242, State: Initialized, Role: FOLLOWER
I20260808 06:00:07.707928 19053 raft_consensus.cc:348] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.708186 19053 raft_consensus.cc:374] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:07.708170 19052 consensus_queue.cc:260] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.708290 19053 raft_consensus.cc:729] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b117e651149409aaecc3468b4295c1c, State: Initialized, Role: FOLLOWER
I20260808 06:00:07.708747 19053 consensus_queue.cc:260] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.709110 19034 heartbeater.cc:503] Master 127.13.229.126:41265 was elected leader, sending a full tablet report...
I20260808 06:00:07.709494 19052 ts_tablet_manager.cc:1428] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260808 06:00:07.709888 19054 raft_consensus.cc:348] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.710093 19054 raft_consensus.cc:374] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:07.710151 19054 raft_consensus.cc:729] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263186d2ae004bd290de9e9e148e88f4, State: Initialized, Role: FOLLOWER
I20260808 06:00:07.710609 19054 consensus_queue.cc:260] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.711032 19053 ts_tablet_manager.cc:1428] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260808 06:00:07.711338 18898 heartbeater.cc:503] Master 127.13.229.126:41265 was elected leader, sending a full tablet report...
I20260808 06:00:07.711339 18761 heartbeater.cc:503] Master 127.13.229.126:41265 was elected leader, sending a full tablet report...
I20260808 06:00:07.711531 19054 ts_tablet_manager.cc:1428] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:00:07.754068 19060 raft_consensus.cc:482] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:07.754316 19060 raft_consensus.cc:504] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.755693 19060 leader_election.cc:288] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b117e651149409aaecc3468b4295c1c (127.13.229.66:45531), e4940ff1600146a1bf4049daf234c242 (127.13.229.67:39735)
I20260808 06:00:07.760186 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.756064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40199 (local address 127.13.229.66:45531)
0808 06:00:07.756224 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.756229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.756510 (+ 281us) server_negotiation.cc:407] Connection header received
0808 06:00:07.756743 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.756746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.756866 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.756958 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:07.757329 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.757920 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.759097 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.759603 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.759696 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.759738 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.759977 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.760051 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:07.760215 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.755946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:45531 (local address 127.13.229.65:40199)
0808 06:00:07.756408 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.756428 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.756597 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.756986 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.756990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.757033 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:07.757210 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.757221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.758050 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.758053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.758969 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.758979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.759724 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.759727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.759826 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.759855 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.759956 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.760029 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"spinlock_wait_cycles":2816,"thread_start_us":194,"threads_started":1}
I20260808 06:00:07.760464 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.756601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:50767 (local address 127.13.229.67:39735)
0808 06:00:07.756756 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.756766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.756995 (+ 229us) server_negotiation.cc:407] Connection header received
0808 06:00:07.757130 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.757133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.757188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.757275 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:07.757572 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.758236 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.759378 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.759838 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.760018 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.760064 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.760243 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.760322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:00:07.760509 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.756503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:39735 (local address 127.13.229.65:50767)
0808 06:00:07.756904 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.756920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:07.757025 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:07.757312 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:07.757315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:07.757334 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:07.757437 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.757443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.758345 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.758348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.759247 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:07.759258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.760007 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.760011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:07.760107 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.760129 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 06:00:07.760210 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 06:00:07.760313 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":1,"spinlock_wait_cycles":7040,"thread_start_us":120,"threads_started":1}
I20260808 06:00:07.760993 18852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d66efba87f54d6cac63157231fd1c50" candidate_uuid: "263186d2ae004bd290de9e9e148e88f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b117e651149409aaecc3468b4295c1c" is_pre_election: true
I20260808 06:00:07.761005 18988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d66efba87f54d6cac63157231fd1c50" candidate_uuid: "263186d2ae004bd290de9e9e148e88f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4940ff1600146a1bf4049daf234c242" is_pre_election: true
I20260808 06:00:07.761252 18988 raft_consensus.cc:2454] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263186d2ae004bd290de9e9e148e88f4 in term 0.
I20260808 06:00:07.761252 18852 raft_consensus.cc:2454] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 263186d2ae004bd290de9e9e148e88f4 in term 0.
I20260808 06:00:07.761732 18644 leader_election.cc:302] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263186d2ae004bd290de9e9e148e88f4, 3b117e651149409aaecc3468b4295c1c; no voters:
I20260808 06:00:07.762024 19060 raft_consensus.cc:2781] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:07.762115 19060 raft_consensus.cc:482] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:07.762159 19060 raft_consensus.cc:3037] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:07.763232 19060 raft_consensus.cc:504] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.763581 19060 leader_election.cc:288] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3b117e651149409aaecc3468b4295c1c (127.13.229.66:45531), e4940ff1600146a1bf4049daf234c242 (127.13.229.67:39735)
I20260808 06:00:07.764039 18988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d66efba87f54d6cac63157231fd1c50" candidate_uuid: "263186d2ae004bd290de9e9e148e88f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4940ff1600146a1bf4049daf234c242"
I20260808 06:00:07.764039 18852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d66efba87f54d6cac63157231fd1c50" candidate_uuid: "263186d2ae004bd290de9e9e148e88f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b117e651149409aaecc3468b4295c1c"
I20260808 06:00:07.764173 18988 raft_consensus.cc:3037] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:07.764173 18852 raft_consensus.cc:3037] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:07.765538 18852 raft_consensus.cc:2454] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263186d2ae004bd290de9e9e148e88f4 in term 1.
I20260808 06:00:07.765538 18988 raft_consensus.cc:2454] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 263186d2ae004bd290de9e9e148e88f4 in term 1.
I20260808 06:00:07.765849 18644 leader_election.cc:302] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263186d2ae004bd290de9e9e148e88f4, e4940ff1600146a1bf4049daf234c242; no voters:
I20260808 06:00:07.766036 19060 raft_consensus.cc:2781] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:07.766275 19060 raft_consensus.cc:686] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 1 LEADER]: Becoming Leader. State: Replica: 263186d2ae004bd290de9e9e148e88f4, State: Running, Role: LEADER
I20260808 06:00:07.766620 19060 consensus_queue.cc:237] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } }
I20260808 06:00:07.769933 18559 catalog_manager.cc:5495] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 263186d2ae004bd290de9e9e148e88f4 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "263186d2ae004bd290de9e9e148e88f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "263186d2ae004bd290de9e9e148e88f4" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 } health_report { overall_health: UNKNOWN } } }
I20260808 06:00:07.783071 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.778995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.13.229.65:36711)
0808 06:00:07.779156 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:07.779160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:07.779174 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:07.779342 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:07.779345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:07.779403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:07.779552 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:07.779830 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.780425 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.781576 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:07.781798 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:07.781904 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:07.782244 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:07.782674 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:07.782812 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:07.782861 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:00:07.782921 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 06:00:07.789182 18852 raft_consensus.cc:1261] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 1 FOLLOWER]: Refusing update from remote peer 263186d2ae004bd290de9e9e148e88f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:07.789235 18988 raft_consensus.cc:1261] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 1 FOLLOWER]: Refusing update from remote peer 263186d2ae004bd290de9e9e148e88f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:07.789929 19060 consensus_queue.cc:1035] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b117e651149409aaecc3468b4295c1c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:07.790172 19063 consensus_queue.cc:1035] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4940ff1600146a1bf4049daf234c242" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 39735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:07.798432 19069 mvcc.cc:205] Tried to move back new op lower bound from 7316147436696563712 to 7316147436615073792. Current Snapshot: MvccSnapshot[applied={T|T < 7316147436696563712}]
I20260808 06:00:07.806811 18561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:07.806996 18561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:07.809399 18561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b368d864f8704644996bd77b671c7e63: Sending DeleteTablet for 3 replicas of tablet 6d66efba87f54d6cac63157231fd1c50
I20260808 06:00:07.810149 18690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d66efba87f54d6cac63157231fd1c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:07 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:37236
I20260808 06:00:07.810385 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d66efba87f54d6cac63157231fd1c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:07 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:51970
I20260808 06:00:07.810736 19080 tablet_replica.cc:330] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4: stopping tablet replica
I20260808 06:00:07.811103 19080 raft_consensus.cc:2229] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:07.811168 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18627
I20260808 06:00:07.811151 18967 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d66efba87f54d6cac63157231fd1c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:07 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:57834
I20260808 06:00:07.811447 19080 raft_consensus.cc:2258] T 6d66efba87f54d6cac63157231fd1c50 P 263186d2ae004bd290de9e9e148e88f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:07.814195 19081 tablet_replica.cc:330] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: stopping tablet replica
I20260808 06:00:07.814510 19081 raft_consensus.cc:2229] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:07.814819 19081 raft_consensus.cc:2258] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:07.815217 19082 tablet_replica.cc:330] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: stopping tablet replica
I20260808 06:00:07.815498 19082 raft_consensus.cc:2229] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:07.815780 19082 raft_consensus.cc:2258] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:07.816533 19081 ts_tablet_manager.cc:1905] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:07.817274 19082 ts_tablet_manager.cc:1905] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:07.819386 19081 ts_tablet_manager.cc:1918] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:07.819496 19081 log.cc:1196] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/6d66efba87f54d6cac63157231fd1c50
I20260808 06:00:07.819844 19081 ts_tablet_manager.cc:1939] T 6d66efba87f54d6cac63157231fd1c50 P 3b117e651149409aaecc3468b4295c1c: Deleting consensus metadata
I20260808 06:00:07.820097 19082 ts_tablet_manager.cc:1918] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:07.820187 19082 log.cc:1196] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/6d66efba87f54d6cac63157231fd1c50
I20260808 06:00:07.820495 19082 ts_tablet_manager.cc:1939] T 6d66efba87f54d6cac63157231fd1c50 P e4940ff1600146a1bf4049daf234c242: Deleting consensus metadata
W20260808 06:00:07.820703 18547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:36711 (error 108)
I20260808 06:00:07.821254 18545 catalog_manager.cc:4841] TS 3b117e651149409aaecc3468b4295c1c (127.13.229.66:45531): tablet 6d66efba87f54d6cac63157231fd1c50 (table test-table [id=64e0d5a6d85040448085bf262c428b08]) successfully deleted
I20260808 06:00:07.821597 18545 catalog_manager.cc:4841] TS e4940ff1600146a1bf4049daf234c242 (127.13.229.67:39735): tablet 6d66efba87f54d6cac63157231fd1c50 (table test-table [id=64e0d5a6d85040448085bf262c428b08]) successfully deleted
I20260808 06:00:07.821862 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:07.821454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:36711 (local address 127.0.0.1:37244)
0808 06:00:07.821692 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:07.821773 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260808 06:00:07.821942 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18766
W20260808 06:00:07.822023 18547 catalog_manager.cc:4584] TS 263186d2ae004bd290de9e9e148e88f4 (127.13.229.65:36711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d66efba87f54d6cac63157231fd1c50: Network error: Client connection negotiation failed: client connection to 127.13.229.65:36711: connect: Connection refused (error 111)
I20260808 06:00:07.829947 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18903
I20260808 06:00:07.837957 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 18529
2026-08-08T06:00:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2327 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T06:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:07Z Disabled control of system clock
I20260808 06:00:07.870817 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45477
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 06:00:07.984484 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:07.984797 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:07.984839 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:07.989063 19087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:07.989130 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:07.989145 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:07.989163 19087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:07.989212 19087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:07.994055 19087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:07.995167 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:07.996680 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:08.005133 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.005339 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.005414 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:08.005549 19087 server_base.cc:1034] running on GCE node
I20260808 06:00:08.006170 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:08.007287 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:08.008476 19087 hybrid_clock.cc:648] HybridClock initialized: now 1786168808008456 us; error 33 us; skew 500 ppm
I20260808 06:00:08.010291 19087 webserver.cc:460] Webserver started at http://127.13.229.126:44975/ using document root <none> and password file <none>
I20260808 06:00:08.010885 19087 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:08.010977 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:08.011216 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:08.013320 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "ad85037652d6462da2649d3f85c45ef5"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "ed6e2da0e299310bf31ab797817a9a3e"
server_key_iv: "cac92cde9baa9f18221ac27c43cf083f"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.013916 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "ad85037652d6462da2649d3f85c45ef5"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "ed6e2da0e299310bf31ab797817a9a3e"
server_key_iv: "cac92cde9baa9f18221ac27c43cf083f"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.017660 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:08.020196 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.021435 19087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:08.021577 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "ad85037652d6462da2649d3f85c45ef5"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "ed6e2da0e299310bf31ab797817a9a3e"
server_key_iv: "cac92cde9baa9f18221ac27c43cf083f"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.021692 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:08.026479 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:08.027112 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:08.027302 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:08.034611 19087 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:45477
I20260808 06:00:08.034642 19154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:45477 every 8 connection(s)
I20260808 06:00:08.035549 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:08.036724 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19087
I20260808 06:00:08.036873 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:08.037171 14229 external_mini_cluster.cc:1428] Setting key c744078ac8b31b21d9309dbdab50b014
I20260808 06:00:08.038899 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:08.043321 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.038516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.13.229.126:45477)
0808 06:00:08.039649 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.039666 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.039703 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:00:08.040412 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.040431 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.040789 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.041085 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.041973 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.041979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.041991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.042002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.042057 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.042073 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.042190 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.042315 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.042590 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":638,"threads_started":1}
I20260808 06:00:08.045518 19155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: Bootstrap starting.
I20260808 06:00:08.047760 19155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:08.048568 19155 log.cc:824] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:08.050464 19155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: No bootstrap required, opened a new log
I20260808 06:00:08.054059 19155 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } }
I20260808 06:00:08.054306 19155 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:08.054386 19155 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad85037652d6462da2649d3f85c45ef5, State: Initialized, Role: FOLLOWER
I20260808 06:00:08.054857 19155 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } }
I20260808 06:00:08.055016 19155 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:08.055104 19155 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:08.055205 19155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:08.056217 19155 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } }
I20260808 06:00:08.056559 19155 leader_election.cc:302] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad85037652d6462da2649d3f85c45ef5; no voters:
I20260808 06:00:08.056879 19155 leader_election.cc:288] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:08.057026 19160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:08.057260 19160 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [term 1 LEADER]: Becoming Leader. State: Replica: ad85037652d6462da2649d3f85c45ef5, State: Running, Role: LEADER
I20260808 06:00:08.057555 19160 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } }
I20260808 06:00:08.058136 19155 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:08.059271 19162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad85037652d6462da2649d3f85c45ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } } }
I20260808 06:00:08.059413 19162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:08.059695 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad85037652d6462da2649d3f85c45ef5. Latest consensus state: current_term: 1 leader_uuid: "ad85037652d6462da2649d3f85c45ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad85037652d6462da2649d3f85c45ef5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45477 } } }
I20260808 06:00:08.059796 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:08.059824 19170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:08.063040 19170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:08.069401 19170 catalog_manager.cc:1347] Generated new cluster ID: b7aa5da2ddf147b89ba5751ebd956222
I20260808 06:00:08.069499 19170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:08.085755 19170 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:08.086122 19170 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:08.094221 19170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: Generated new TSK 0
I20260808 06:00:08.094918 19170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:08.101648 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--builtin_ntp_servers=127.13.229.84:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 06:00:08.220538 19180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:08.220814 19180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:08.220860 19180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:08.224784 19180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:08.224835 19180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:08.224890 19180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:08.229846 19180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:08.230922 19180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:08.232354 19180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:08.242010 19188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.242064 19185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:08.242062 19180 server_base.cc:1034] running on GCE node
W20260808 06:00:08.242192 19186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:08.242691 19180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:08.243336 19180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:08.244526 19180 hybrid_clock.cc:648] HybridClock initialized: now 1786168808244513 us; error 32 us; skew 500 ppm
I20260808 06:00:08.246500 19180 webserver.cc:460] Webserver started at http://127.13.229.65:41405/ using document root <none> and password file <none>
I20260808 06:00:08.247097 19180 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:08.247200 19180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:08.247440 19180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:08.249590 19180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "ae683dc3969f4319802c680fd70c1d74"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "996bab8221a8285c93775b65811eac32"
server_key_iv: "141310bab0df0dd391dec4e31b2da811"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.250231 19180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "ae683dc3969f4319802c680fd70c1d74"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "996bab8221a8285c93775b65811eac32"
server_key_iv: "141310bab0df0dd391dec4e31b2da811"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.254060 19180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 06:00:08.256649 19195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.257898 19180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:00:08.258044 19180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "ae683dc3969f4319802c680fd70c1d74"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "996bab8221a8285c93775b65811eac32"
server_key_iv: "141310bab0df0dd391dec4e31b2da811"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.258157 19180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:08.263016 19180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:08.263630 19180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:08.263823 19180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:08.264400 19180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:08.265318 19180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:08.265386 19180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.265452 19180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:08.265506 19180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.274416 19180 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:34729
I20260808 06:00:08.274441 19311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:34729 every 8 connection(s)
I20260808 06:00:08.275377 19180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:08.279093 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19180
I20260808 06:00:08.279219 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:08.279553 14229 external_mini_cluster.cc:1428] Setting key b34181a80b820276b95d714fab348618
I20260808 06:00:08.281623 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--builtin_ntp_servers=127.13.229.84:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 06:00:08.282073 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.277478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40447 (local address 127.13.229.126:45477)
0808 06:00:08.277628 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.277632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.278523 (+ 891us) server_negotiation.cc:407] Connection header received
0808 06:00:08.280332 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.280336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.280381 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.280463 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.281591 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.281595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.281598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.281610 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.281630 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.281645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.281735 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.281889 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.281942 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:08.283010 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.277684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45477 (local address 127.13.229.65:40447)
0808 06:00:08.278351 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.278392 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.280113 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.280626 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.280633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.280722 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.281321 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.281323 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.281363 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.281440 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.281735 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.281747 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.281756 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.281937 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.282269 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260808 06:00:08.284425 19312 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45477
I20260808 06:00:08.284664 19312 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:08.285085 19312 heartbeater.cc:511] Master 127.13.229.126:45477 requested a full tablet report, sending...
I20260808 06:00:08.286867 19119 ts_manager.cc:194] Registered new tserver with Master: ae683dc3969f4319802c680fd70c1d74 (127.13.229.65:34729)
W20260808 06:00:08.400096 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:08.400357 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:08.400396 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:08.404363 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:08.404412 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:08.404465 19315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:08.409089 19315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:08.410066 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:08.411439 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:08.421022 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.421037 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.421022 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:08.421337 19315 server_base.cc:1034] running on GCE node
I20260808 06:00:08.421743 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:08.422310 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:08.423477 19315 hybrid_clock.cc:648] HybridClock initialized: now 1786168808423462 us; error 27 us; skew 500 ppm
I20260808 06:00:08.425504 19315 webserver.cc:460] Webserver started at http://127.13.229.66:46793/ using document root <none> and password file <none>
I20260808 06:00:08.426192 19315 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:08.426283 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:08.426501 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:08.428620 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "728e77c8ffe3411b8071bfb3767eb21c"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "f393e231e32f0d178066640c3e4515fe"
server_key_iv: "cdf761a6399748ba57a9bc3100292167"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.429246 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "728e77c8ffe3411b8071bfb3767eb21c"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "f393e231e32f0d178066640c3e4515fe"
server_key_iv: "cdf761a6399748ba57a9bc3100292167"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.433000 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:08.435575 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.436856 19315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:08.437000 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "728e77c8ffe3411b8071bfb3767eb21c"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "f393e231e32f0d178066640c3e4515fe"
server_key_iv: "cdf761a6399748ba57a9bc3100292167"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.437117 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:08.441819 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:08.442448 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:08.442631 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:08.443174 19315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:08.444134 19315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:08.444204 19315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.444272 19315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:08.444314 19315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.453012 19315 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:46639
I20260808 06:00:08.453042 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:46639 every 8 connection(s)
I20260808 06:00:08.453928 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:08.459394 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19315
I20260808 06:00:08.459532 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:08.459928 14229 external_mini_cluster.cc:1428] Setting key d9b9c81bc905273daa4c4e26146f3fd4
I20260808 06:00:08.461037 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.455722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:33187 (local address 127.13.229.126:45477)
0808 06:00:08.455877 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.455880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.456949 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 06:00:08.458899 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.458917 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.458969 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.459079 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.460562 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.460566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.460569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.460583 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.460607 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.460618 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.460713 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.460896 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.460934 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 06:00:08.462049 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--builtin_ntp_servers=127.13.229.84:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 06:00:08.462136 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.456145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45477 (local address 127.13.229.66:33187)
0808 06:00:08.456800 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.456841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.458561 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.459288 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.459297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.459425 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.460218 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.460219 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.460260 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.460355 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.460733 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.460745 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.460751 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.460933 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.461180 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260808 06:00:08.463402 19449 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45477
I20260808 06:00:08.463658 19449 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:08.464180 19449 heartbeater.cc:511] Master 127.13.229.126:45477 requested a full tablet report, sending...
I20260808 06:00:08.465242 19119 ts_manager.cc:194] Registered new tserver with Master: 728e77c8ffe3411b8071bfb3767eb21c (127.13.229.66:46639)
W20260808 06:00:08.578543 19452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:08.578810 19452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:08.578850 19452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:08.582948 19452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:08.583009 19452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:08.583097 19452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:08.587769 19452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:08.588897 19452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:08.590436 19452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:08.600251 19457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.600247 19460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:08.600303 19458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:08.600533 19452 server_base.cc:1034] running on GCE node
I20260808 06:00:08.601085 19452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:08.601727 19452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:08.602898 19452 hybrid_clock.cc:648] HybridClock initialized: now 1786168808602864 us; error 45 us; skew 500 ppm
I20260808 06:00:08.604802 19452 webserver.cc:460] Webserver started at http://127.13.229.67:44711/ using document root <none> and password file <none>
I20260808 06:00:08.605419 19452 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:08.605489 19452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:08.605747 19452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:08.607792 19452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "bb6e4788a69e41d7a2d275d6816f7611"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "8183aecb722051eac55a54dd6d5e8b78"
server_key_iv: "ef2f71d4eeb85742b2ebb6acf7d94c60"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.608438 19452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "bb6e4788a69e41d7a2d275d6816f7611"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "8183aecb722051eac55a54dd6d5e8b78"
server_key_iv: "ef2f71d4eeb85742b2ebb6acf7d94c60"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.612267 19452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 06:00:08.614886 19467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.616405 19452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:08.616529 19452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "bb6e4788a69e41d7a2d275d6816f7611"
format_stamp: "Formatted at 2026-08-08 06:00:08 on dist-test-slave-zkpd"
server_key: "8183aecb722051eac55a54dd6d5e8b78"
server_key_iv: "ef2f71d4eeb85742b2ebb6acf7d94c60"
server_key_version: "encryptionkey@0"
I20260808 06:00:08.616644 19452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:08.621995 19452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:08.622677 19452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:08.622886 19452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:08.623483 19452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:08.624486 19452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:08.624534 19452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.624598 19452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:08.624630 19452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:08.634318 19452 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:33131
I20260808 06:00:08.634347 19589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:33131 every 8 connection(s)
I20260808 06:00:08.635284 19452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:08.639750 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19452
I20260808 06:00:08.639894 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:08.640264 14229 external_mini_cluster.cc:1428] Setting key aba984e1580a7bc0ef707ef74774a152
I20260808 06:00:08.642001 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.637224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:36671 (local address 127.13.229.126:45477)
0808 06:00:08.637381 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.637386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.638505 (+ 1119us) server_negotiation.cc:407] Connection header received
0808 06:00:08.639920 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.639924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.639974 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.640086 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.641505 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.641508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.641511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.641525 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.641547 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.641557 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.641657 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.641804 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.641868 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:00:08.643076 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.637645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45477 (local address 127.13.229.67:36671)
0808 06:00:08.638309 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.638350 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.639658 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.640394 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.640403 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.640509 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.641178 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.641182 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.641242 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.641337 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.641648 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.641656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.641660 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.641870 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.642136 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260808 06:00:08.644145 19590 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45477
I20260808 06:00:08.644397 19590 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:08.644814 19590 heartbeater.cc:511] Master 127.13.229.126:45477 requested a full tablet report, sending...
I20260808 06:00:08.645805 19119 ts_manager.cc:194] Registered new tserver with Master: bb6e4788a69e41d7a2d275d6816f7611 (127.13.229.67:33131)
I20260808 06:00:08.654436 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:00:08.658152 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.656446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.13.229.126:45477)
0808 06:00:08.656608 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.656614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.656782 (+ 168us) server_negotiation.cc:407] Connection header received
0808 06:00:08.656946 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.656949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.656989 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.657076 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.657737 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.657741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.657743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.657763 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.657784 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.657799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.657889 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.657976 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.658018 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:00:08.661109 19119 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:08.664371 19119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:08.676085 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.672956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34729 (local address 127.0.0.1:54312)
0808 06:00:08.673853 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.673867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.673937 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.674566 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.674572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.674648 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.675274 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.675275 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.675292 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.675347 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.675692 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.675703 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.675710 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.675832 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.675943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":131,"threads_started":1}
I20260808 06:00:08.676128 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.673689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.13.229.65:34729)
0808 06:00:08.674142 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.674155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.674189 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:00:08.674247 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.674253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.674420 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.674572 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.675478 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.675485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.675499 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.675513 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.675559 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.675592 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.675698 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.675867 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.675970 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":124,"threads_started":1}
I20260808 06:00:08.676465 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.672865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:46639 (local address 127.0.0.1:41440)
0808 06:00:08.673487 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.673535 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.673640 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.674593 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.674596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.674648 (+ 52us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.675332 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.675334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.675347 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.675378 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.676229 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.676234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.676239 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.676306 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.676339 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":103,"threads_started":1}
I20260808 06:00:08.677403 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.672990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.13.229.66:46639)
0808 06:00:08.674211 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.674224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.674248 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:00:08.674301 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.674306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.674461 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.674608 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.676023 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.676029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.676039 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.676048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.676084 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.676110 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.676579 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.676629 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.677238 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":112,"threads_started":1}
I20260808 06:00:08.677843 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.673204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33131 (local address 127.0.0.1:41086)
0808 06:00:08.675442 (+ 2238us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.675457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.675540 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.676582 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.676584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.676596 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.677106 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.677109 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.677120 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.677151 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.677667 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.677669 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.677672 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.677728 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.677761 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2161,"thread_start_us":99,"threads_started":1}
I20260808 06:00:08.677976 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.673697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.13.229.67:33131)
0808 06:00:08.674207 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.674224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.675554 (+ 1330us) server_negotiation.cc:407] Connection header received
0808 06:00:08.675620 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.675629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.675832 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.675984 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.677290 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.677295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.677306 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.677317 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.677359 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.677393 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.677691 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.677754 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.677816 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":171,"threads_started":1}
I20260808 06:00:08.680457 19380 tablet_service.cc:1467] Processing CreateTablet for tablet c2a718d08e8447ae8482bed17a778c00 (DEFAULT_TABLE table=test-table [id=4a1c079ec9e04d65ab61bf95b71c2d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:08.680382 19245 tablet_service.cc:1467] Processing CreateTablet for tablet c2a718d08e8447ae8482bed17a778c00 (DEFAULT_TABLE table=test-table [id=4a1c079ec9e04d65ab61bf95b71c2d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:08.680382 19519 tablet_service.cc:1467] Processing CreateTablet for tablet c2a718d08e8447ae8482bed17a778c00 (DEFAULT_TABLE table=test-table [id=4a1c079ec9e04d65ab61bf95b71c2d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:08.681440 19380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a718d08e8447ae8482bed17a778c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:08.681464 19245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a718d08e8447ae8482bed17a778c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:08.681464 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a718d08e8447ae8482bed17a778c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:08.687554 19609 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Bootstrap starting.
I20260808 06:00:08.689446 19611 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Bootstrap starting.
I20260808 06:00:08.690469 19609 tablet_bootstrap.cc:654] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:08.691212 19609 log.cc:824] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:08.691298 19611 tablet_bootstrap.cc:654] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:08.692101 19611 log.cc:824] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:08.692272 19610 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Bootstrap starting.
I20260808 06:00:08.693317 19609 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: No bootstrap required, opened a new log
I20260808 06:00:08.693531 19609 ts_tablet_manager.cc:1397] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:08.694203 19611 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: No bootstrap required, opened a new log
I20260808 06:00:08.694394 19611 ts_tablet_manager.cc:1397] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260808 06:00:08.694712 19610 tablet_bootstrap.cc:654] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:08.695529 19610 log.cc:824] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:08.697729 19610 tablet_bootstrap.cc:492] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: No bootstrap required, opened a new log
I20260808 06:00:08.697647 19609 raft_consensus.cc:348] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.697922 19609 raft_consensus.cc:374] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:08.697937 19610 ts_tablet_manager.cc:1397] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:08.697970 19609 raft_consensus.cc:729] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728e77c8ffe3411b8071bfb3767eb21c, State: Initialized, Role: FOLLOWER
I20260808 06:00:08.698431 19609 consensus_queue.cc:260] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.698812 19611 raft_consensus.cc:348] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.699069 19611 raft_consensus.cc:374] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:08.699136 19611 raft_consensus.cc:729] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4788a69e41d7a2d275d6816f7611, State: Initialized, Role: FOLLOWER
I20260808 06:00:08.699129 19449 heartbeater.cc:503] Master 127.13.229.126:45477 was elected leader, sending a full tablet report...
I20260808 06:00:08.699560 19611 consensus_queue.cc:260] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.700249 19590 heartbeater.cc:503] Master 127.13.229.126:45477 was elected leader, sending a full tablet report...
I20260808 06:00:08.700645 19611 ts_tablet_manager.cc:1428] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260808 06:00:08.702404 19609 ts_tablet_manager.cc:1428] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260808 06:00:08.702487 19610 raft_consensus.cc:348] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.702759 19610 raft_consensus.cc:374] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:08.702833 19610 raft_consensus.cc:729] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae683dc3969f4319802c680fd70c1d74, State: Initialized, Role: FOLLOWER
I20260808 06:00:08.703331 19610 consensus_queue.cc:260] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.704067 19312 heartbeater.cc:503] Master 127.13.229.126:45477 was elected leader, sending a full tablet report...
I20260808 06:00:08.704448 19610 ts_tablet_manager.cc:1428] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260808 06:00:08.704799 19450 tablet.cc:2369] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:00:08.771093 19615 raft_consensus.cc:482] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:08.771337 19615 raft_consensus.cc:504] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.772614 19615 leader_election.cc:288] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae683dc3969f4319802c680fd70c1d74 (127.13.229.65:34729), bb6e4788a69e41d7a2d275d6816f7611 (127.13.229.67:33131)
I20260808 06:00:08.774484 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.772812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34729 (local address 127.13.229.66:49505)
0808 06:00:08.772945 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.772964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.773050 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.773318 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.773324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.773340 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.773883 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.773885 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.773895 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.773922 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.774215 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.774218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.774222 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.774297 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.774331 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 06:00:08.774523 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.772909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:49505 (local address 127.13.229.65:34729)
0808 06:00:08.773054 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.773059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.773077 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:00:08.773133 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.773138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.773182 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.773279 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.774059 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.774065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.774072 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.774084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.774100 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.774109 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.774214 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.774326 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.774375 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 06:00:08.774652 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.773253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:41891 (local address 127.13.229.67:33131)
0808 06:00:08.773404 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.773407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.773417 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:00:08.773453 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.773455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.773483 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.773544 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.774219 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.774224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.774230 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.774247 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.774307 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.774319 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.774410 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.774489 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.774524 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 06:00:08.775018 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.772880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33131 (local address 127.13.229.66:41891)
0808 06:00:08.773190 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.773200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:08.773284 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:08.773570 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:08.773574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:08.773589 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:08.774057 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:08.774058 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:08.774068 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:08.774094 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:08.774409 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:08.774412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:08.774416 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:00:08.774494 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 06:00:08.774893 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":133,"threads_started":1}
I20260808 06:00:08.775198 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a718d08e8447ae8482bed17a778c00" candidate_uuid: "728e77c8ffe3411b8071bfb3767eb21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae683dc3969f4319802c680fd70c1d74" is_pre_election: true
I20260808 06:00:08.775552 19265 raft_consensus.cc:2454] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728e77c8ffe3411b8071bfb3767eb21c in term 0.
I20260808 06:00:08.775583 19539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a718d08e8447ae8482bed17a778c00" candidate_uuid: "728e77c8ffe3411b8071bfb3767eb21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4788a69e41d7a2d275d6816f7611" is_pre_election: true
I20260808 06:00:08.775866 19539 raft_consensus.cc:2454] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 728e77c8ffe3411b8071bfb3767eb21c in term 0.
I20260808 06:00:08.776115 19332 leader_election.cc:302] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728e77c8ffe3411b8071bfb3767eb21c, ae683dc3969f4319802c680fd70c1d74; no voters:
I20260808 06:00:08.776409 19615 raft_consensus.cc:2781] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:08.776489 19615 raft_consensus.cc:482] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:08.776518 19615 raft_consensus.cc:3037] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 0 FOLLOWER]: Advancing to term 1
W20260808 06:00:08.776652 19313 tablet.cc:2369] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:00:08.777644 19615 raft_consensus.cc:504] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.778024 19615 leader_election.cc:288] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [CANDIDATE]: Term 1 election: Requested vote from peers ae683dc3969f4319802c680fd70c1d74 (127.13.229.65:34729), bb6e4788a69e41d7a2d275d6816f7611 (127.13.229.67:33131)
I20260808 06:00:08.778430 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a718d08e8447ae8482bed17a778c00" candidate_uuid: "728e77c8ffe3411b8071bfb3767eb21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae683dc3969f4319802c680fd70c1d74"
I20260808 06:00:08.778591 19265 raft_consensus.cc:3037] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:08.778594 19539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a718d08e8447ae8482bed17a778c00" candidate_uuid: "728e77c8ffe3411b8071bfb3767eb21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4788a69e41d7a2d275d6816f7611"
I20260808 06:00:08.778738 19539 raft_consensus.cc:3037] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:08.779774 19265 raft_consensus.cc:2454] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728e77c8ffe3411b8071bfb3767eb21c in term 1.
I20260808 06:00:08.779857 19539 raft_consensus.cc:2454] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 728e77c8ffe3411b8071bfb3767eb21c in term 1.
I20260808 06:00:08.780113 19332 leader_election.cc:302] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728e77c8ffe3411b8071bfb3767eb21c, ae683dc3969f4319802c680fd70c1d74; no voters:
I20260808 06:00:08.780334 19615 raft_consensus.cc:2781] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:08.780575 19615 raft_consensus.cc:686] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 1 LEADER]: Becoming Leader. State: Replica: 728e77c8ffe3411b8071bfb3767eb21c, State: Running, Role: LEADER
I20260808 06:00:08.780917 19615 consensus_queue.cc:237] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } }
I20260808 06:00:08.784355 19119 catalog_manager.cc:5495] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c reported cstate change: term changed from 0 to 1, leader changed from <none> to 728e77c8ffe3411b8071bfb3767eb21c (127.13.229.66). New cstate: current_term: 1 leader_uuid: "728e77c8ffe3411b8071bfb3767eb21c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "728e77c8ffe3411b8071bfb3767eb21c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 } health_report { overall_health: UNKNOWN } } }
I20260808 06:00:08.798454 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.797098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.13.229.66:46639)
0808 06:00:08.797217 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:08.797220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:08.797233 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:08.797349 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:08.797351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:08.797421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:08.797496 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:08.798135 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:08.798138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:08.798140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:08.798151 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:08.798169 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:08.798177 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:08.798255 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:08.798316 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 06:00:08.798348 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 06:00:08.804203 19265 raft_consensus.cc:1261] T c2a718d08e8447ae8482bed17a778c00 P ae683dc3969f4319802c680fd70c1d74 [term 1 FOLLOWER]: Refusing update from remote peer 728e77c8ffe3411b8071bfb3767eb21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:08.804203 19539 raft_consensus.cc:1261] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 1 FOLLOWER]: Refusing update from remote peer 728e77c8ffe3411b8071bfb3767eb21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:08.804881 19619 consensus_queue.cc:1035] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae683dc3969f4319802c680fd70c1d74" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:08.805066 19615 consensus_queue.cc:1035] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4788a69e41d7a2d275d6816f7611" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:08.813073 19624 mvcc.cc:205] Tried to move back new op lower bound from 7316147440854872064 to 7316147440769150976. Current Snapshot: MvccSnapshot[applied={T|T < 7316147440854872064}]
I20260808 06:00:08.821141 19119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:08.821311 19119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:08.824062 19119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad85037652d6462da2649d3f85c45ef5: Sending DeleteTablet for 3 replicas of tablet c2a718d08e8447ae8482bed17a778c00
I20260808 06:00:08.824925 19380 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a718d08e8447ae8482bed17a778c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:08 UTC) from {username='slave'} at 127.0.0.1:41440
I20260808 06:00:08.825027 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a718d08e8447ae8482bed17a778c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:08 UTC) from {username='slave'} at 127.0.0.1:41086
I20260808 06:00:08.825461 19245 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a718d08e8447ae8482bed17a778c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:08 UTC) from {username='slave'} at 127.0.0.1:54312
I20260808 06:00:08.825917 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19180
I20260808 06:00:08.826012 19633 tablet_replica.cc:330] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: stopping tablet replica
I20260808 06:00:08.825932 19632 tablet_replica.cc:330] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: stopping tablet replica
I20260808 06:00:08.827267 19632 raft_consensus.cc:2229] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:08.827582 19632 raft_consensus.cc:2258] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:08.828032 19633 raft_consensus.cc:2229] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:08.828286 19633 raft_consensus.cc:2258] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:08.829751 19633 ts_tablet_manager.cc:1905] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:08.829751 19632 ts_tablet_manager.cc:1905] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:08.832520 19633 ts_tablet_manager.cc:1918] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:08.832520 19632 ts_tablet_manager.cc:1918] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:08.832628 19632 log.cc:1196] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/c2a718d08e8447ae8482bed17a778c00
I20260808 06:00:08.832628 19633 log.cc:1196] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/c2a718d08e8447ae8482bed17a778c00
I20260808 06:00:08.832970 19632 ts_tablet_manager.cc:1939] T c2a718d08e8447ae8482bed17a778c00 P 728e77c8ffe3411b8071bfb3767eb21c: Deleting consensus metadata
I20260808 06:00:08.832970 19633 ts_tablet_manager.cc:1939] T c2a718d08e8447ae8482bed17a778c00 P bb6e4788a69e41d7a2d275d6816f7611: Deleting consensus metadata
I20260808 06:00:08.834193 19106 catalog_manager.cc:4841] TS bb6e4788a69e41d7a2d275d6816f7611 (127.13.229.67:33131): tablet c2a718d08e8447ae8482bed17a778c00 (table test-table [id=4a1c079ec9e04d65ab61bf95b71c2d35]) successfully deleted
I20260808 06:00:08.834193 19104 catalog_manager.cc:4841] TS 728e77c8ffe3411b8071bfb3767eb21c (127.13.229.66:46639): tablet c2a718d08e8447ae8482bed17a778c00 (table test-table [id=4a1c079ec9e04d65ab61bf95b71c2d35]) successfully deleted
W20260808 06:00:08.837973 19103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:34729 (error 108)
I20260808 06:00:08.838912 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19315
I20260808 06:00:08.839119 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:08.838722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34729 (local address 127.0.0.1:54314)
0808 06:00:08.838929 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:08.839014 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260808 06:00:08.840626 19103 catalog_manager.cc:4584] TS ae683dc3969f4319802c680fd70c1d74 (127.13.229.65:34729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2a718d08e8447ae8482bed17a778c00: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34729: connect: Connection refused (error 111)
I20260808 06:00:08.846485 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19452
I20260808 06:00:08.853497 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19087
2026-08-08T06:00:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1014 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 06:00:08.876207 14229 mini_oidc.cc:152] Starting JWKS server
I20260808 06:00:08.877828 14229 webserver.cc:460] Webserver started at https://127.0.0.1:42783/ using document root <none> and password file <none>
I20260808 06:00:08.878019 14229 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:00:08.878309 14229 webserver.cc:460] Webserver started at http://127.0.0.1:41873/ using document root <none> and password file <none>
2026-08-08T06:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:08Z Disabled control of system clock
I20260808 06:00:08.888496 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42379
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:36017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:42379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/testchainca.cert with env {}
W20260808 06:00:09.002480 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:09.002754 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:09.002795 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:09.006944 19642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:09.007007 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:09.007023 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:09.007040 19642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:09.007081 19642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:00:09.007117 19642 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:00:09.007135 19642 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42783/jwks.json
I20260808 06:00:09.012527 19642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:42379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:09.013631 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:09.014750 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:09.023345 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:09.023325 19642 server_base.cc:1034] running on GCE node
W20260808 06:00:09.023329 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:09.023312 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:09.024290 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:09.025449 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:09.026640 19642 hybrid_clock.cc:648] HybridClock initialized: now 1786168809026630 us; error 33 us; skew 500 ppm
I20260808 06:00:09.028662 19642 webserver.cc:460] Webserver started at http://127.13.229.126:32919/ using document root <none> and password file <none>
I20260808 06:00:09.029271 19642 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:09.029368 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:09.029577 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:09.031648 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "532404f6a3b745c2b8eb51f18d806826"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "9b4859cf6c2a3f9c5092a50aab013107"
server_key_iv: "1dd4cc4a4d42f823ab5af131c29b0b5d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.032301 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "532404f6a3b745c2b8eb51f18d806826"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "9b4859cf6c2a3f9c5092a50aab013107"
server_key_iv: "1dd4cc4a4d42f823ab5af131c29b0b5d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.036087 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:00:09.038554 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:09.039809 19642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:09.039987 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "532404f6a3b745c2b8eb51f18d806826"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "9b4859cf6c2a3f9c5092a50aab013107"
server_key_iv: "1dd4cc4a4d42f823ab5af131c29b0b5d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.040105 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:09.065258 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:09.066064 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:09.066288 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:09.073813 19642 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:42379
I20260808 06:00:09.073844 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:42379 every 8 connection(s)
I20260808 06:00:09.074774 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:09.075212 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19642
I20260808 06:00:09.075361 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:09.075654 14229 external_mini_cluster.cc:1428] Setting key b16273e5460015b67ab88f20812b1b2d
I20260808 06:00:09.079207 19713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:09.087019 19713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826: Bootstrap starting.
I20260808 06:00:09.087136 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.077034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.13.229.126:42379)
0808 06:00:09.077783 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.077805 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.077855 (+ 50us) server_negotiation.cc:407] Connection header received
0808 06:00:09.078813 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.078843 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.079395 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.079803 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:09.080316 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.082000 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.082905 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.083663 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.083944 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.084911 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:09.084919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:09.084937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:09.085479 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:09.085555 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:09.085588 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:09.085707 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:09.085826 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 06:00:09.086129 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260808 06:00:09.089975 19713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:09.090407 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:00:09.091025 19713 log.cc:824] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:09.093161 19713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826: No bootstrap required, opened a new log
I20260808 06:00:09.096918 19713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } }
I20260808 06:00:09.097168 19713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:09.097259 19713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532404f6a3b745c2b8eb51f18d806826, State: Initialized, Role: FOLLOWER
I20260808 06:00:09.097743 19713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } }
I20260808 06:00:09.097934 19713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:09.098004 19713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:09.098099 19713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:09.099025 19713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } }
I20260808 06:00:09.099378 19713 leader_election.cc:302] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 532404f6a3b745c2b8eb51f18d806826; no voters:
I20260808 06:00:09.099712 19713 leader_election.cc:288] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:09.099833 19718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:09.100112 19718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [term 1 LEADER]: Becoming Leader. State: Replica: 532404f6a3b745c2b8eb51f18d806826, State: Running, Role: LEADER
I20260808 06:00:09.100418 19718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } }
I20260808 06:00:09.101013 19713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:09.102113 19721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532404f6a3b745c2b8eb51f18d806826. Latest consensus state: current_term: 1 leader_uuid: "532404f6a3b745c2b8eb51f18d806826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } } }
I20260808 06:00:09.102268 19721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.102691 19729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:09.102648 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "532404f6a3b745c2b8eb51f18d806826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532404f6a3b745c2b8eb51f18d806826" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42379 } } }
I20260808 06:00:09.103441 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.105964 19729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:09.111650 19729 catalog_manager.cc:1347] Generated new cluster ID: 0779179c430745ec8a1e808f9495311a
I20260808 06:00:09.111730 19729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:09.123008 19729 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:09.123999 19729 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:09.131815 19729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 532404f6a3b745c2b8eb51f18d806826: Generated new TSK 0
I20260808 06:00:09.132565 19729 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 06:00:09.158432 19746 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 06:00:09.158572 19746 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 06:00:09.159271 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.141428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.13.229.126:42379)
0808 06:00:09.141562 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.141567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.141734 (+ 167us) server_negotiation.cc:407] Connection header received
0808 06:00:09.142004 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.142008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.142067 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.142177 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:00:09.142468 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.143287 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.144383 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.144735 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.144841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.145144 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:00:09.158834 (+ 13690us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 06:00:09.159126 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55534: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 06:00:09.159376 19716 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55534: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 06:00:09.160584 19745 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:09.141301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42379 (local address 127.0.0.1:55534)
0808 06:00:09.141657 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:09.141670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:09.141827 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:09.142237 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:09.142241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:09.142249 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:00:09.142353 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.142359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.143421 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.143424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.144264 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.144274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.144849 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.144852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.144916 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.144963 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:00:09.159103 (+ 14140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 06:00:09.159317 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.229.126:42379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.229.126:42379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 06:00:09.161303 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19642
2026-08-08T06:00:09Z chronyd exiting
I20260808 06:00:09.201658 14229 test_util.cc:175] -----------------------------------------------
I20260808 06:00:09.201748 14229 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786168783448266-14229-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 06:00:09.202952 14229 mini_oidc.cc:152] Starting JWKS server
I20260808 06:00:09.203848 14229 webserver.cc:460] Webserver started at https://127.0.0.1:41341/ using document root <none> and password file <none>
I20260808 06:00:09.204023 14229 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:00:09.204313 14229 webserver.cc:460] Webserver started at http://127.0.0.1:38233/ using document root <none> and password file <none>
2026-08-08T06:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:09Z Disabled control of system clock
I20260808 06:00:09.214118 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43523
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:44643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:43523
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/testchainca.cert with env {}
W20260808 06:00:09.327389 19752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:09.327661 19752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:09.327701 19752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:09.331662 19752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:09.331720 19752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:09.331735 19752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:09.331753 19752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:09.331768 19752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:00:09.331832 19752 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:00:09.331852 19752 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41341/jwks.json
I20260808 06:00:09.337822 19752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:43523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:09.339234 19752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:09.340374 19752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:09.348702 19757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:09.348752 19752 server_base.cc:1034] running on GCE node
W20260808 06:00:09.348731 19760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:09.348702 19758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:09.349618 19752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:09.350811 19752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:09.351974 19752 hybrid_clock.cc:648] HybridClock initialized: now 1786168809351954 us; error 31 us; skew 500 ppm
I20260808 06:00:09.353924 19752 webserver.cc:460] Webserver started at http://127.13.229.126:37725/ using document root <none> and password file <none>
I20260808 06:00:09.354507 19752 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:09.354570 19752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:09.354815 19752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:09.356948 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "acf401006ebb48f89c0cb95d637840e8"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "75b2c9a30ef979feeee4ff4d41d0c22c"
server_key_iv: "746a139584637c94200c6b8067249cf0"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.357527 19752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "acf401006ebb48f89c0cb95d637840e8"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "75b2c9a30ef979feeee4ff4d41d0c22c"
server_key_iv: "746a139584637c94200c6b8067249cf0"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.361196 19752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:09.363590 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:09.364912 19752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:09.365062 19752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "acf401006ebb48f89c0cb95d637840e8"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "75b2c9a30ef979feeee4ff4d41d0c22c"
server_key_iv: "746a139584637c94200c6b8067249cf0"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.365180 19752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:09.376298 19752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:09.376947 19752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:09.377137 19752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:09.384115 19752 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:43523
I20260808 06:00:09.384124 19822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:43523 every 8 connection(s)
I20260808 06:00:09.385059 19752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:09.388353 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:09.390275 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19752
I20260808 06:00:09.390419 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:09.390781 14229 external_mini_cluster.cc:1428] Setting key 5f98e38924d353d4c4ced5676bfae806
I20260808 06:00:09.394791 19823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8: Bootstrap starting.
I20260808 06:00:09.397142 19823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:09.397925 19823 log.cc:824] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:09.400054 19823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8: No bootstrap required, opened a new log
I20260808 06:00:09.400586 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.392353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.13.229.126:43523)
0808 06:00:09.393020 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.393035 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.393072 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:00:09.393900 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.393921 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.394328 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.394701 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:09.395164 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.396206 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.397135 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.397668 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.397849 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.398737 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:09.398742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:09.398757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:09.399144 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:09.399199 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:09.399220 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:09.399332 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:09.399454 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 06:00:09.399724 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260808 06:00:09.403604 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:00:09.404343 19823 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } }
I20260808 06:00:09.404641 19823 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:09.404719 19823 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf401006ebb48f89c0cb95d637840e8, State: Initialized, Role: FOLLOWER
I20260808 06:00:09.405191 19823 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } }
I20260808 06:00:09.405349 19823 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:09.405436 19823 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:09.405536 19823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:09.406450 19823 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } }
I20260808 06:00:09.406801 19823 leader_election.cc:302] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf401006ebb48f89c0cb95d637840e8; no voters:
I20260808 06:00:09.407142 19823 leader_election.cc:288] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:09.407217 19828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:09.407405 19828 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [term 1 LEADER]: Becoming Leader. State: Replica: acf401006ebb48f89c0cb95d637840e8, State: Running, Role: LEADER
I20260808 06:00:09.407696 19828 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } }
I20260808 06:00:09.408469 19823 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:09.409505 19830 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf401006ebb48f89c0cb95d637840e8. Latest consensus state: current_term: 1 leader_uuid: "acf401006ebb48f89c0cb95d637840e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } } }
I20260808 06:00:09.409538 19829 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf401006ebb48f89c0cb95d637840e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf401006ebb48f89c0cb95d637840e8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43523 } } }
I20260808 06:00:09.409627 19830 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.409657 19829 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.410084 19837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:09.414032 19837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:09.419585 19837 catalog_manager.cc:1347] Generated new cluster ID: 3beeb825d21f4a1fafa2e6107f582756
I20260808 06:00:09.419656 19837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:09.435374 19837 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:09.436678 19837 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:09.451118 19837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf401006ebb48f89c0cb95d637840e8: Generated new TSK 0
I20260808 06:00:09.451936 19837 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 06:00:09.490725 19853 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 06:00:09.490885 19853 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 06:00:09.491643 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.467747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.13.229.126:43523)
0808 06:00:09.468438 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.468443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.468459 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:09.468561 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.468565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.468628 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.468771 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:00:09.469047 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.469602 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.470596 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.470851 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.470969 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.471311 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:00:09.491147 (+ 19836us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 06:00:09.491462 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":588}
W20260808 06:00:09.491724 19826 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 06:00:09.491770 19852 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:09.467651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43523 (local address 127.0.0.1:35280)
0808 06:00:09.468110 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:09.468130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:09.468269 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:09.468766 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:09.468770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:09.468778 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:00:09.468928 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.468935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.469730 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.469735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.470467 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.470477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.470986 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.470990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.471081 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.471113 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:00:09.491379 (+ 20266us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 06:00:09.491579 (+ 200us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.229.126:43523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260808 06:00:09.492448 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19752
2026-08-08T06:00:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 06:00:09.533830 14229 mini_oidc.cc:152] Starting JWKS server
I20260808 06:00:09.534663 14229 webserver.cc:460] Webserver started at https://127.0.0.1:34857/ using document root <none> and password file <none>
I20260808 06:00:09.534744 14229 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 06:00:09.535012 14229 webserver.cc:460] Webserver started at http://127.0.0.1:35391/ using document root <none> and password file <none>
2026-08-08T06:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:09Z Disabled control of system clock
I20260808 06:00:09.545251 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:44787
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:44787
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34857/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 06:00:09.658402 19859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:09.658804 19859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:09.658924 19859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:09.663183 19859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:09.663285 19859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:09.663316 19859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:09.663357 19859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:09.663393 19859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:00:09.663425 19859 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:00:09.663456 19859 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34857/jwks.json
I20260808 06:00:09.668288 19859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:44787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34857/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:09.669471 19859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:09.670934 19859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:09.679579 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:09.679626 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:09.679597 19869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:09.680024 19859 server_base.cc:1034] running on GCE node
I20260808 06:00:09.680696 19859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:09.682093 19859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:09.683259 19859 hybrid_clock.cc:648] HybridClock initialized: now 1786168809683238 us; error 31 us; skew 500 ppm
I20260808 06:00:09.685211 19859 webserver.cc:460] Webserver started at http://127.13.229.126:44477/ using document root <none> and password file <none>
I20260808 06:00:09.685792 19859 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:09.685889 19859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:09.686141 19859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:09.688303 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "cfe5ac31d3e047379907fd6c86ba2ba5"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "46a3ac09764e27443d4cc3dda353da86"
server_key_iv: "ad9ccda12156a2ab26c445adafa0152d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.688890 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "cfe5ac31d3e047379907fd6c86ba2ba5"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "46a3ac09764e27443d4cc3dda353da86"
server_key_iv: "ad9ccda12156a2ab26c445adafa0152d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.692761 19859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:00:09.695199 19876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:09.696539 19859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:09.696677 19859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "cfe5ac31d3e047379907fd6c86ba2ba5"
format_stamp: "Formatted at 2026-08-08 06:00:09 on dist-test-slave-zkpd"
server_key: "46a3ac09764e27443d4cc3dda353da86"
server_key_iv: "ad9ccda12156a2ab26c445adafa0152d"
server_key_version: "encryptionkey@0"
I20260808 06:00:09.696789 19859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:09.722019 19859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:09.722777 19859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:09.722981 19859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:09.730168 19859 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:44787
I20260808 06:00:09.730177 19930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:44787 every 8 connection(s)
I20260808 06:00:09.731134 19859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:09.734324 19931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:09.739866 19931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5: Bootstrap starting.
I20260808 06:00:09.741405 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19859
I20260808 06:00:09.741528 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:09.741824 14229 external_mini_cluster.cc:1428] Setting key 6c8986235c640d6e1766e9f78979f0ac
I20260808 06:00:09.742873 19931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:09.743662 19931 log.cc:824] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:09.745877 19931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5: No bootstrap required, opened a new log
I20260808 06:00:09.750260 19931 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } }
I20260808 06:00:09.750516 19931 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:09.750604 19931 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe5ac31d3e047379907fd6c86ba2ba5, State: Initialized, Role: FOLLOWER
I20260808 06:00:09.750934 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.742885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.13.229.126:44787)
0808 06:00:09.743552 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.743567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.743604 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:00:09.744320 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.744339 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.744670 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.744955 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:09.745284 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.746289 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.747506 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.748037 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.748162 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.749006 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:09.749011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:09.749024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:09.749409 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:09.749465 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:09.749484 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:09.749562 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:09.749717 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 06:00:09.750015 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260808 06:00:09.751093 19931 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } }
I20260808 06:00:09.751264 19931 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:09.751335 19931 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:09.751432 19931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:09.752460 19931 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } }
I20260808 06:00:09.752909 19931 leader_election.cc:302] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfe5ac31d3e047379907fd6c86ba2ba5; no voters:
I20260808 06:00:09.753404 19931 leader_election.cc:288] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:09.753492 19936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:09.753752 19936 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cfe5ac31d3e047379907fd6c86ba2ba5, State: Running, Role: LEADER
I20260808 06:00:09.754065 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:00:09.754092 19936 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } }
I20260808 06:00:09.754791 19931 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:09.755214 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfe5ac31d3e047379907fd6c86ba2ba5. Latest consensus state: current_term: 1 leader_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } } }
I20260808 06:00:09.755331 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.756032 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfe5ac31d3e047379907fd6c86ba2ba5" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 44787 } } }
I20260808 06:00:09.756134 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:09.756393 19940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:09.759449 19940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:09.765333 19940 catalog_manager.cc:1347] Generated new cluster ID: e4ed328aeaa9473f875e1ebb24c2737d
I20260808 06:00:09.765412 19940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:09.790591 19940 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:09.791599 19940 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:09.795573 19940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfe5ac31d3e047379907fd6c86ba2ba5: Generated new TSK 0
I20260808 06:00:09.796411 19940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:09.819466 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:09.814240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.13.229.126:44787)
0808 06:00:09.814400 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:09.814404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:09.814541 (+ 137us) server_negotiation.cc:407] Connection header received
0808 06:00:09.814687 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:09.814691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:09.814750 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:09.814876 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:00:09.815088 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.815974 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.816827 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.817181 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.817290 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.817565 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:00:09.819053 (+ 1488us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 06:00:09.819297 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 06:00:09.819545 19934 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 06:00:09.821116 19961 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:09.814120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:44787 (local address 127.0.0.1:37722)
0808 06:00:09.814456 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:09.814470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:09.814567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:09.814886 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:09.814888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:09.814894 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:00:09.814985 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.814989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.816077 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.816080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.816717 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:09.816724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:09.817296 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:09.817299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:09.817362 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:09.817389 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:00:09.819227 (+ 1838us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 06:00:09.820948 (+ 1721us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.229.126:44787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1519,"thread_start_us":112,"threads_started":1}
I20260808 06:00:09.821816 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19859
E20260808 06:00:09.833114 19963 webserver.cc:533] Webserver: sslize failed:
E20260808 06:00:09.833189 19963 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T06:00:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T06:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:09Z Disabled control of system clock
I20260808 06:00:09.874481 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:39747
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:39747
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/testchainca.cert with env {}
W20260808 06:00:09.988852 19967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:09.989116 19967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:09.989157 19967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:09.993395 19967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:09.993460 19967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:09.993476 19967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:09.993494 19967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:09.993542 19967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 06:00:09.993571 19967 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 06:00:09.993605 19967 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 06:00:09.999049 19967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:39747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:10.000262 19967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:10.001322 19967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:10.009811 19972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:10.009821 19975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:10.010046 19973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:10.010167 19967 server_base.cc:1034] running on GCE node
I20260808 06:00:10.010855 19967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:10.012025 19967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:10.013216 19967 hybrid_clock.cc:648] HybridClock initialized: now 1786168810013203 us; error 28 us; skew 500 ppm
I20260808 06:00:10.015105 19967 webserver.cc:460] Webserver started at http://127.13.229.126:39839/ using document root <none> and password file <none>
I20260808 06:00:10.015698 19967 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:10.015790 19967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:10.016052 19967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:10.018105 19967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7"
format_stamp: "Formatted at 2026-08-08 06:00:10 on dist-test-slave-zkpd"
server_key: "202f9e8fcd720dc49930e1bffea67397"
server_key_iv: "a2416efeffd2b4098e7dcf6d4aea9dfa"
server_key_version: "encryptionkey@0"
I20260808 06:00:10.018695 19967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7"
format_stamp: "Formatted at 2026-08-08 06:00:10 on dist-test-slave-zkpd"
server_key: "202f9e8fcd720dc49930e1bffea67397"
server_key_iv: "a2416efeffd2b4098e7dcf6d4aea9dfa"
server_key_version: "encryptionkey@0"
I20260808 06:00:10.022444 19967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 06:00:10.025022 19981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:10.026358 19967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:10.026525 19967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7"
format_stamp: "Formatted at 2026-08-08 06:00:10 on dist-test-slave-zkpd"
server_key: "202f9e8fcd720dc49930e1bffea67397"
server_key_iv: "a2416efeffd2b4098e7dcf6d4aea9dfa"
server_key_version: "encryptionkey@0"
I20260808 06:00:10.026654 19967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:10.067503 19967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:10.068365 19967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:10.068590 19967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:10.077258 19967 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:39747
I20260808 06:00:10.077275 20036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:39747 every 8 connection(s)
I20260808 06:00:10.078786 19967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:10.081141 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 19967
I20260808 06:00:10.081283 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:10.081586 14229 external_mini_cluster.cc:1428] Setting key 0a05b4a5e75827eeb31acb95d48c59bd
I20260808 06:00:10.083811 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:10.089607 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7: Bootstrap starting.
I20260808 06:00:10.091387 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:10.082891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.13.229.126:39747)
0808 06:00:10.083526 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:10.083542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:10.083578 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:00:10.084326 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:10.084348 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:10.084759 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:10.085120 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:10.085570 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.086549 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.087516 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.088074 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.088247 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:10.089202 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:10.089206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:10.089220 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:10.089612 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:10.089668 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:10.089688 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:10.089812 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:10.090012 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 06:00:10.090305 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260808 06:00:10.092566 20037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:10.093374 20037 log.cc:824] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:10.094678 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:00:10.096048 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7: No bootstrap required, opened a new log
I20260808 06:00:10.099576 20037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } }
I20260808 06:00:10.099797 20037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:10.099841 20037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d736e23f365455e9c6f6fd7c5bf9dd7, State: Initialized, Role: FOLLOWER
I20260808 06:00:10.100291 20037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } }
I20260808 06:00:10.100416 20037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:10.100464 20037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:10.100559 20037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:10.101464 20037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } }
I20260808 06:00:10.101809 20037 leader_election.cc:302] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d736e23f365455e9c6f6fd7c5bf9dd7; no voters:
I20260808 06:00:10.102272 20037 leader_election.cc:288] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:10.102418 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:10.102679 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 7d736e23f365455e9c6f6fd7c5bf9dd7, State: Running, Role: LEADER
I20260808 06:00:10.103031 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } }
I20260808 06:00:10.103524 20037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:10.104832 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } } }
I20260808 06:00:10.104831 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d736e23f365455e9c6f6fd7c5bf9dd7. Latest consensus state: current_term: 1 leader_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d736e23f365455e9c6f6fd7c5bf9dd7" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39747 } } }
I20260808 06:00:10.105086 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:10.105116 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:10.106125 20052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:10.109383 20052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:10.115355 20052 catalog_manager.cc:1347] Generated new cluster ID: 391e59bb769e48a885f049007d391664
I20260808 06:00:10.115428 20052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:10.138890 20052 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:10.139981 20052 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:10.143817 20052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d736e23f365455e9c6f6fd7c5bf9dd7: Generated new TSK 0
I20260808 06:00:10.144567 20052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:10.187321 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:10.158995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.13.229.126:39747)
0808 06:00:10.159152 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:10.159159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:10.159324 (+ 165us) server_negotiation.cc:407] Connection header received
0808 06:00:10.159507 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:10.159510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:10.159561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:10.159647 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0808 06:00:10.159990 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.160527 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.161799 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.162005 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.162079 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:10.162437 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 06:00:10.186880 (+ 24443us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 06:00:10.187182 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55980: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 06:00:10.187438 20040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55980: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 06:00:10.187942 20066 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:10.158918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:39747 (local address 127.0.0.1:55980)
0808 06:00:10.159226 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:10.159242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:10.159362 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:10.159673 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:10.159676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:10.159690 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 06:00:10.159819 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:10.159826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.160645 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.160649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:10.161664 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:10.161674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:10.162133 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:10.162137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:10.162235 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:10.162264 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 06:00:10.187149 (+ 24885us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 06:00:10.187377 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.229.126:39747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":116,"threads_started":1}
I20260808 06:00:10.188580 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 19967
2026-08-08T06:00:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (344 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:10 dist-test-slave-zkpd krb5kdc[20078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:10 dist-test-slave-zkpd krb5kdc[20078](info): set up 2 sockets
Aug 08 06:00:10 dist-test-slave-zkpd krb5kdc[20078](info): commencing operation
krb5kdc: starting...
W20260808 06:00:11.615376 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 06:00:11.920171 14229 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.305s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 06:00:12.192807 14229 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.272s user 0.001s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 06:00:12.394419 14229 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.201s user 0.001s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/test-user.keytab.
W20260808 06:00:12.779309 14229 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.385s user 0.000s sys 0.002s
Aug 08 06:00:12 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:12.838642 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43511
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:12.975654 20093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:12.976023 20093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:12.976116 20093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:12.980152 20093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:12.980259 20093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:12.980316 20093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:12.980376 20093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:12.980415 20093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:12.985873 20093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:12.987093 20093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:12.988255 20093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:12.996994 20099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:12.997009 20101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:12.997071 20093 server_base.cc:1034] running on GCE node
W20260808 06:00:12.997009 20098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:12.997808 20093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:12.998854 20093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:12.999976 20093 hybrid_clock.cc:648] HybridClock initialized: now 1786168812999941 us; error 43 us; skew 500 ppm
Aug 08 06:00:12 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168812, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.002678 20093 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:13.003557 20093 webserver.cc:460] Webserver started at http://127.13.229.126:34345/ using document root <none> and password file <none>
I20260808 06:00:13.004158 20093 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:13.004223 20093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:13.004402 20093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:13.006419 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "b324fa0eff9a45fab0236288062cb286"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "7c16c4e82597fcba3e42fc6f1e73c69e"
server_key_iv: "ee103d3a3ecc70ed194aa2aa1bdd6499"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.006940 20093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "b324fa0eff9a45fab0236288062cb286"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "7c16c4e82597fcba3e42fc6f1e73c69e"
server_key_iv: "ee103d3a3ecc70ed194aa2aa1bdd6499"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.010722 20093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 06:00:13.013260 20109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.014485 20093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:13.014590 20093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "b324fa0eff9a45fab0236288062cb286"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "7c16c4e82597fcba3e42fc6f1e73c69e"
server_key_iv: "ee103d3a3ecc70ed194aa2aa1bdd6499"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.014678 20093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:13.034688 20093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:13.038167 20093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:13.038339 20093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:13.045856 20093 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:43511
I20260808 06:00:13.045879 20162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:43511 every 8 connection(s)
I20260808 06:00:13.046875 20093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:13.049780 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:13.050969 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20093
I20260808 06:00:13.051110 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:13.051419 14229 external_mini_cluster.cc:1428] Setting key 563ceec20fbdd6901468d6453459ecb4
I20260808 06:00:13.055876 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: Bootstrap starting.
I20260808 06:00:13.058102 20163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:13.059005 20163 log.cc:824] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:13.061045 20163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: No bootstrap required, opened a new log
I20260808 06:00:13.064792 20163 raft_consensus.cc:348] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } }
I20260808 06:00:13.065088 20163 raft_consensus.cc:374] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:13.065158 20163 raft_consensus.cc:729] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b324fa0eff9a45fab0236288062cb286, State: Initialized, Role: FOLLOWER
I20260808 06:00:13.065690 20163 consensus_queue.cc:260] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } }
I20260808 06:00:13.065819 20163 raft_consensus.cc:388] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:13.065908 20163 raft_consensus.cc:482] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:13.066031 20163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:13.066190 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.052953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.13.229.126:43511)
0808 06:00:13.053425 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.053440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.053478 (+ 38us) server_negotiation.cc:407] Connection header received
0808 06:00:13.054140 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.054158 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.054498 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.054840 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.055688 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.056762 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.057620 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.058091 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.058280 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.061059 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.061065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.061079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.061106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.062976 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.063484 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.063490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.063495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.063611 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.063932 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.063939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.063943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.064366 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.064591 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.064855 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.065218 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260808 06:00:13.067005 20163 raft_consensus.cc:504] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } }
I20260808 06:00:13.067368 20163 leader_election.cc:302] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b324fa0eff9a45fab0236288062cb286; no voters:
I20260808 06:00:13.067737 20163 leader_election.cc:288] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:13.067821 20169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:13.068058 20169 raft_consensus.cc:686] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [term 1 LEADER]: Becoming Leader. State: Replica: b324fa0eff9a45fab0236288062cb286, State: Running, Role: LEADER
I20260808 06:00:13.068353 20169 consensus_queue.cc:237] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } }
I20260808 06:00:13.069057 20163 sys_catalog.cc:564] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:13.069535 20169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b324fa0eff9a45fab0236288062cb286. Latest consensus state: current_term: 1 leader_uuid: "b324fa0eff9a45fab0236288062cb286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } } }
I20260808 06:00:13.069650 20169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:13.070055 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b324fa0eff9a45fab0236288062cb286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b324fa0eff9a45fab0236288062cb286" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43511 } } }
I20260808 06:00:13.070161 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286 [sys.catalog]: This master's current role is: LEADER
W20260808 06:00:13.074379 20186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:00:13.074450 20186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:00:13.074534 20174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:13.075201 20174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:13.080915 20174 catalog_manager.cc:1347] Generated new cluster ID: a38e30c76e444e18a8c76eefcdcfad17
I20260808 06:00:13.081008 20174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:13.092796 20174 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:13.094352 20174 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:13.099964 20174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b324fa0eff9a45fab0236288062cb286: Generated new TSK 0
I20260808 06:00:13.100821 20174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:13.137005 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:13.252653 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:13.252897 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:13.252940 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:13.257041 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:13.257107 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:13.257218 20194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:13.262441 20194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:13.263554 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:13.264652 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:13.274641 20200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.274715 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.274672 20199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:13.274842 20194 server_base.cc:1034] running on GCE node
I20260808 06:00:13.275300 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:13.275997 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:13.277182 20194 hybrid_clock.cc:648] HybridClock initialized: now 1786168813277181 us; error 40 us; skew 500 ppm
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.279959 20194 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:13.280807 20194 webserver.cc:460] Webserver started at http://127.13.229.65:44689/ using document root <none> and password file <none>
I20260808 06:00:13.281373 20194 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:13.281464 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:13.281702 20194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:13.283756 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "9939cd9ef91b4d7eaa01a64e310277d9"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "b7ba8b3bb465622a5ed4474119639079"
server_key_iv: "40b434a21ce705c5ff0848370e033395"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.284425 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "9939cd9ef91b4d7eaa01a64e310277d9"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "b7ba8b3bb465622a5ed4474119639079"
server_key_iv: "40b434a21ce705c5ff0848370e033395"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.288154 20194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:00:13.290622 20209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.292004 20194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:13.292145 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "9939cd9ef91b4d7eaa01a64e310277d9"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "b7ba8b3bb465622a5ed4474119639079"
server_key_iv: "40b434a21ce705c5ff0848370e033395"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.292263 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:13.337867 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:13.341403 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:13.341617 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:13.342245 20194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:13.343096 20194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:13.343165 20194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.343240 20194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:13.343292 20194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.352880 20194 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:39261
I20260808 06:00:13.353611 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:39261 every 8 connection(s)
I20260808 06:00:13.353945 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:13.354708 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20194
I20260808 06:00:13.354846 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:13.355140 14229 external_mini_cluster.cc:1428] Setting key 9d90a1119e4f480074fe6d6b3349ba53
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:13.370038 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.356914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:51851 (local address 127.13.229.126:43511)
0808 06:00:13.359030 (+ 2116us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.359036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.359057 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:00:13.359487 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.359493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.359554 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.359656 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.360915 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.361859 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.363060 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.363503 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.363640 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.366179 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.366182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.366185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.366209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.367647 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.368321 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.368324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.368326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.368409 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.368834 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.368838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.368839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.369078 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.369188 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.369729 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.369857 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996}
I20260808 06:00:13.371084 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.357501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43511 (local address 127.13.229.65:51851)
0808 06:00:13.358033 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:13.358076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:13.359275 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:13.359835 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:13.359841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:13.360256 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:13.360760 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.360774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.362012 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.362019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.362925 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.362935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.363660 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.363663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.363806 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.364510 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:13.364526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:13.365996 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:13.367768 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.367774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.367789 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.368178 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.368506 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.368510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.368512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.368739 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.369186 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:13.369193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:13.369465 (+ 272us) client_negotiation.cc:769] Sending connection context
0808 06:00:13.369649 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 06:00:13.369987 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
I20260808 06:00:13.372480 20328 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43511
I20260808 06:00:13.372743 20328 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:13.373240 20328 heartbeater.cc:511] Master 127.13.229.126:43511 requested a full tablet report, sending...
I20260808 06:00:13.374769 20126 ts_manager.cc:194] Registered new tserver with Master: 9939cd9ef91b4d7eaa01a64e310277d9 (127.13.229.65:39261)
I20260808 06:00:13.375952 20126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:51851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:13.385263 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:13.500509 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:13.500792 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:13.500887 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:13.504925 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:13.504980 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:13.505071 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:13.510216 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:13.511420 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:13.512545 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:13.522228 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.522245 20339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.522241 20342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:13.522583 20333 server_base.cc:1034] running on GCE node
I20260808 06:00:13.523005 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:13.523684 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:13.524885 20333 hybrid_clock.cc:648] HybridClock initialized: now 1786168813524867 us; error 67 us; skew 500 ppm
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.527590 20333 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:13.528489 20333 webserver.cc:460] Webserver started at http://127.13.229.66:33301/ using document root <none> and password file <none>
I20260808 06:00:13.529072 20333 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:13.529186 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:13.529436 20333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:13.531436 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "5117a14769b7433e944a8edf56d60473"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "546f1e6d364b82e8dcc741be71a23cde"
server_key_iv: "c03f1928e8439bd3e6e2d32010a5836d"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.532063 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "5117a14769b7433e944a8edf56d60473"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "546f1e6d364b82e8dcc741be71a23cde"
server_key_iv: "c03f1928e8439bd3e6e2d32010a5836d"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.535698 20333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:13.538125 20349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.539319 20333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:13.539456 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "5117a14769b7433e944a8edf56d60473"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "546f1e6d364b82e8dcc741be71a23cde"
server_key_iv: "c03f1928e8439bd3e6e2d32010a5836d"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.539566 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:13.556905 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:13.559864 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:13.560092 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:13.560657 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:13.561476 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:13.561524 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.561563 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:13.561579 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.570696 20333 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:37005
I20260808 06:00:13.570719 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:37005 every 8 connection(s)
I20260808 06:00:13.571661 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:13.572151 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20333
I20260808 06:00:13.572302 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:13.572616 14229 external_mini_cluster.cc:1428] Setting key 7e4534471c61a8c2f6ed6b945b8816f4
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:00:13.587699 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.573684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:50567 (local address 127.13.229.126:43511)
0808 06:00:13.573837 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.573841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.574946 (+ 1105us) server_negotiation.cc:407] Connection header received
0808 06:00:13.576878 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.576884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.576948 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.577049 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.578434 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.578951 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.580872 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.581074 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.581154 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.584022 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.584030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.584032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.584055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.585491 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.586098 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.586104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.586110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.586185 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.586612 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.586616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.586618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.586797 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.586908 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.587407 (+ 499us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.587525 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 06:00:13.588706 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.574228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43511 (local address 127.13.229.66:50567)
0808 06:00:13.574752 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:13.574823 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:13.576661 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:13.577204 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:13.577210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:13.577562 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:13.578276 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.578299 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.579090 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.579097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.580715 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.580732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.581238 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.581244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.581508 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.582138 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:13.582166 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:13.583796 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:13.585614 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.585620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.585635 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.585981 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.586345 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.586350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.586356 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.586503 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.586911 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:13.586920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:13.587159 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 06:00:13.587337 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 06:00:13.587648 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260808 06:00:13.589921 20472 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43511
I20260808 06:00:13.590184 20472 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:13.590730 20472 heartbeater.cc:511] Master 127.13.229.126:43511 requested a full tablet report, sending...
I20260808 06:00:13.591815 20126 ts_manager.cc:194] Registered new tserver with Master: 5117a14769b7433e944a8edf56d60473 (127.13.229.66:37005)
I20260808 06:00:13.592432 20126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:50567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:13.602473 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:13.716060 20478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:13.716349 20478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:13.716393 20478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:13.720697 20478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:13.720759 20478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:13.720821 20478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:13.726042 20478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:13.727049 20478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:13.728060 20478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:13.737344 20487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.737353 20484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:13.737353 20483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:13.737540 20478 server_base.cc:1034] running on GCE node
I20260808 06:00:13.738122 20478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:13.739044 20478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:13.740267 20478 hybrid_clock.cc:648] HybridClock initialized: now 1786168813740255 us; error 30 us; skew 500 ppm
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.743069 20478 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:13.744045 20478 webserver.cc:460] Webserver started at http://127.13.229.67:38501/ using document root <none> and password file <none>
I20260808 06:00:13.744612 20478 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:13.744680 20478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:13.744851 20478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:13.746825 20478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "7e031cc3df7c4e6d8b2bcc5edae1fedf"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "ebbebb6367e4ac20a8af9fc17dc233af"
server_key_iv: "ab1e17367c1395b888574bf83ecd896f"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.747342 20478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "7e031cc3df7c4e6d8b2bcc5edae1fedf"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "ebbebb6367e4ac20a8af9fc17dc233af"
server_key_iv: "ab1e17367c1395b888574bf83ecd896f"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.750952 20478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:00:13.753271 20494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.754477 20478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:13.754572 20478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "7e031cc3df7c4e6d8b2bcc5edae1fedf"
format_stamp: "Formatted at 2026-08-08 06:00:13 on dist-test-slave-zkpd"
server_key: "ebbebb6367e4ac20a8af9fc17dc233af"
server_key_iv: "ab1e17367c1395b888574bf83ecd896f"
server_key_version: "encryptionkey@0"
I20260808 06:00:13.754652 20478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:13.771366 20478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:13.774718 20478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:13.774937 20478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:13.775538 20478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:13.776387 20478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:13.776434 20478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.776505 20478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:13.776546 20478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:13.785290 20478 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:35421
I20260808 06:00:13.785318 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:35421 every 8 connection(s)
I20260808 06:00:13.786216 20478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:13.789474 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20478
I20260808 06:00:13.789609 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:13.789897 14229 external_mini_cluster.cc:1428] Setting key c19491494dce860a8285b5eb57e81985
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:13.800462 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.788107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35381 (local address 127.13.229.126:43511)
0808 06:00:13.788256 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.788260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.789163 (+ 903us) server_negotiation.cc:407] Connection header received
0808 06:00:13.790790 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.790794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.790849 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.790919 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.792325 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.792850 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.794020 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.794359 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.794477 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.797089 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.797093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.797096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.797125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.798346 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.798861 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.798865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.798867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.798952 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.799308 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.799311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.799313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.799497 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.799606 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.800162 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.800290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:00:13.801337 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.788549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43511 (local address 127.13.229.67:35381)
0808 06:00:13.789014 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:13.789058 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:13.790558 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:13.791088 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:13.791095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:13.791480 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:13.792174 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.792191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.792977 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.792980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.793891 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:13.793901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.794519 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.794523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:13.794694 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.795286 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:13.795302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:13.796928 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:13.798467 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.798473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.798487 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.798742 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.799049 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:13.799052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:13.799054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:13.799194 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:13.799605 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:13.799612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:13.799860 (+ 248us) client_negotiation.cc:769] Sending connection context
0808 06:00:13.800102 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 06:00:13.800418 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260808 06:00:13.802462 20613 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43511
I20260808 06:00:13.802704 20613 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:13.803259 20613 heartbeater.cc:511] Master 127.13.229.126:43511 requested a full tablet report, sending...
I20260808 06:00:13.804404 20126 ts_manager.cc:194] Registered new tserver with Master: 7e031cc3df7c4e6d8b2bcc5edae1fedf (127.13.229.67:35421)
I20260808 06:00:13.804960 20126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:35381
I20260808 06:00:13.815603 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/test-admin.keytab.
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.833357 14229 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 06:00:13.836229 20621 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:13.834765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.13.229.65:39261)
0808 06:00:13.835098 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.835113 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.835138 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:00:13.835188 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.835193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.835384 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.835509 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.836046 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55842: BlockingRecv error: recv got EOF from 127.0.0.1:55842 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":125,"threads_started":1}
W20260808 06:00:13.836266 14229 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.837311 14229 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.838212 14229 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 06:00:13.848040 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.839050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.13.229.65:39261)
0808 06:00:13.839178 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.839182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.839293 (+ 111us) server_negotiation.cc:407] Connection header received
0808 06:00:13.839451 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.839454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.839493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.839575 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.840182 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.840971 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.841829 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.842200 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.842346 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.844678 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.844687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.844700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.844724 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.846358 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.846755 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.846760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.846761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.846837 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.847075 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.847077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.847078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.847301 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.847446 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.847613 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.847814 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 06:00:13.851646 20621 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:13.850810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.13.229.65:39261)
0808 06:00:13.850938 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.850941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.850974 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:00:13.851075 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.851077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.851109 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.851170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.851542 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 06:00:13.851754 14229 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.852761 14229 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.853616 14229 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 06:00:13.861903 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.854400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.13.229.65:39261)
0808 06:00:13.854510 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.854514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.854637 (+ 123us) server_negotiation.cc:407] Connection header received
0808 06:00:13.854721 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.854723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.854755 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.854825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.855276 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.855787 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.856621 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.856854 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.856931 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.859075 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.859081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.859087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.859109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.860605 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.860963 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.860965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.860966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.861033 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.861255 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.861257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.861258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.861404 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.861496 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.861681 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.861799 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 06:00:13.864677 20621 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:13.863633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.13.229.65:39261)
0808 06:00:13.863745 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.863749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.863887 (+ 138us) server_negotiation.cc:407] Connection header received
0808 06:00:13.864045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.864047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.864080 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.864143 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.864575 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55888: BlockingRecv error: recv got EOF from 127.0.0.1:55888 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 06:00:13.864758 14229 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786168783448266-14229-0/krb5kdc/krb5cc' of type FILE
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.865667 14229 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:13.866515 14229 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 06:00:13.875313 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:13.867316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.13.229.65:39261)
0808 06:00:13.867424 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:13.867428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:13.867575 (+ 147us) server_negotiation.cc:407] Connection header received
0808 06:00:13.867728 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:13.867731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:13.867767 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:13.867863 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:13.868365 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.868988 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.869896 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:13.870136 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:13.870240 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:13.872523 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:13.872527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:13.872529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:13.872553 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:13.873760 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.874162 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.874166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.874168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.874252 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:13.874526 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:13.874530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:13.874531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:13.874690 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:13.874820 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:13.875052 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 06:00:13.875151 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:00:13.876289 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20194
I20260808 06:00:13.884665 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20333
I20260808 06:00:13.892343 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20478
I20260808 06:00:13.899699 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20093
2026-08-08T06:00:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3450 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20634](info): set up 2 sockets
Aug 08 06:00:13 dist-test-slave-zkpd krb5kdc[20634](info): commencing operation
krb5kdc: starting...
W20260808 06:00:14.966375 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab.
Entry for principal oryx/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab.
I20260808 06:00:15.092684 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33945
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab
--principal=oryx/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:15.205364 20649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:15.205598 20649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:15.205637 20649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:15.209472 20649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:15.209517 20649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:15.209537 20649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:15.209553 20649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:15.209568 20649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:15.215121 20649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab
--principal=oryx/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:15.216068 20649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:15.217020 20649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:15.225263 20656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.225212 20658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.225421 20655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:15.225353 20649 server_base.cc:1034] running on GCE node
I20260808 06:00:15.226097 20649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:15.227180 20649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:15.228374 20649 hybrid_clock.cc:648] HybridClock initialized: now 1786168815228345 us; error 51 us; skew 500 ppm
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:15.231122 20649 init.cc:377] Logged in from keytab as oryx/127.13.229.126@KRBTEST.COM (short username oryx)
I20260808 06:00:15.232111 20649 webserver.cc:460] Webserver started at http://127.13.229.126:34097/ using document root <none> and password file <none>
I20260808 06:00:15.232671 20649 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:15.232764 20649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:15.233001 20649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:15.235039 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "79feb995c79adb11996cf623b7ac1277"
server_key_iv: "1daa33ac0b2340f525daff3102302d50"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.235603 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "79feb995c79adb11996cf623b7ac1277"
server_key_iv: "1daa33ac0b2340f525daff3102302d50"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.239267 20649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:00:15.241654 20665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.242861 20649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:15.243019 20649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "79feb995c79adb11996cf623b7ac1277"
server_key_iv: "1daa33ac0b2340f525daff3102302d50"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.243132 20649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:15.282362 20649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:15.285632 20649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:15.285866 20649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:15.293416 20649 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:33945
I20260808 06:00:15.293722 20721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:33945 every 8 connection(s)
I20260808 06:00:15.294449 20649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:15.297200 20722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:15.299468 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20649
I20260808 06:00:15.299621 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:15.299964 14229 external_mini_cluster.cc:1428] Setting key 53d493bfedb0f13bb346dc099d86385d
I20260808 06:00:15.303879 20722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Bootstrap starting.
I20260808 06:00:15.306596 20722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:15.307329 20722 log.cc:824] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:15.309720 20722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: No bootstrap required, opened a new log
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:15.313054 20722 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } }
I20260808 06:00:15.313304 20722 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:15.313372 20722 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb01fc67f68e440e95e9f6d0bf4cd8ff, State: Initialized, Role: FOLLOWER
I20260808 06:00:15.313846 20722 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } }
I20260808 06:00:15.314021 20722 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:15.314102 20722 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:15.314229 20722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:15.315408 20722 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } }
I20260808 06:00:15.315797 20722 leader_election.cc:302] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb01fc67f68e440e95e9f6d0bf4cd8ff; no voters:
I20260808 06:00:15.315882 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:15.301449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.13.229.126:33945)
0808 06:00:15.301882 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:15.301909 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:15.301951 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:00:15.302739 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:15.302762 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:15.303211 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:15.303574 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:15.304443 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.305902 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.306829 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.307314 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.307500 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:15.310352 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:15.310357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:15.310376 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:15.310403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:15.312599 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.313080 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.313085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.313089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.313202 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.313520 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.313524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.313526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.314010 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:15.314210 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:15.314556 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 06:00:15.314857 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260808 06:00:15.316918 20722 leader_election.cc:288] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:15.317024 20727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:15.317233 20727 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 LEADER]: Becoming Leader. State: Replica: bb01fc67f68e440e95e9f6d0bf4cd8ff, State: Running, Role: LEADER
I20260808 06:00:15.317592 20727 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } }
I20260808 06:00:15.318270 20722 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:15.318879 20729 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb01fc67f68e440e95e9f6d0bf4cd8ff. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } }
I20260808 06:00:15.319005 20729 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:15.319294 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } }
I20260808 06:00:15.319469 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:15.319516 20733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:15.322777 20733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:15.328320 20733 catalog_manager.cc:1347] Generated new cluster ID: ae1d44125a27488caafaddc63bca4f1b
I20260808 06:00:15.328395 20733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:15.337750 20733 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:15.338615 20733 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:15.344457 20733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Generated new TSK 0
I20260808 06:00:15.345080 20733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
I20260808 06:00:15.381048 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:15.512178 20748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:15.512427 20748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:15.512468 20748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:15.516394 20748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:15.516441 20748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:15.516496 20748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:15.521726 20748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:15.522678 20748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:15.523679 20748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:15.533205 20748 server_base.cc:1034] running on GCE node
W20260808 06:00:15.533227 20754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.533229 20753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.533183 20756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:15.533804 20748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:15.534425 20748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:15.535610 20748 hybrid_clock.cc:648] HybridClock initialized: now 1786168815535576 us; error 44 us; skew 500 ppm
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:15.538440 20748 init.cc:377] Logged in from keytab as oryx/127.13.229.65@KRBTEST.COM (short username oryx)
I20260808 06:00:15.539259 20748 webserver.cc:460] Webserver started at http://127.13.229.65:38959/ using document root <none> and password file <none>
I20260808 06:00:15.539815 20748 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:15.539925 20748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:15.540160 20748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:15.542104 20748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "73778dd88e2a417e86297b0a56f16ddc"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "44087f2c09ca6e5f83f016346f3f8349"
server_key_iv: "cd02838198fbf17050fd4c030856d4ff"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.542668 20748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "73778dd88e2a417e86297b0a56f16ddc"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "44087f2c09ca6e5f83f016346f3f8349"
server_key_iv: "cd02838198fbf17050fd4c030856d4ff"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.546411 20748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 06:00:15.548823 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.550005 20748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:15.550141 20748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "73778dd88e2a417e86297b0a56f16ddc"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "44087f2c09ca6e5f83f016346f3f8349"
server_key_iv: "cd02838198fbf17050fd4c030856d4ff"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.550258 20748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:15.565963 20748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:15.569371 20748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:15.569583 20748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:15.570184 20748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:15.571049 20748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:15.571144 20748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.571219 20748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:15.571262 20748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.579804 20748 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:41229
I20260808 06:00:15.579831 20881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:41229 every 8 connection(s)
I20260808 06:00:15.580662 20748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:15.588148 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20748
I20260808 06:00:15.588297 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:15.588624 14229 external_mini_cluster.cc:1428] Setting key 6e22550623e04475a9da3c1e4515a963
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:15.594264 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:15.582351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:53365 (local address 127.13.229.126:33945)
0808 06:00:15.582523 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:15.582528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:15.583339 (+ 811us) server_negotiation.cc:407] Connection header received
0808 06:00:15.584701 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:15.584703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:15.584750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:15.584826 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:15.585917 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.586448 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.587526 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.587743 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.587821 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:15.590561 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:15.590564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:15.590566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:15.590589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:15.592186 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.592754 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.592758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.592760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.592844 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.593185 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.593190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.593192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.593368 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:15.593481 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:15.593969 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 06:00:15.594094 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 06:00:15.595086 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:15.582741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:53365)
0808 06:00:15.583194 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:15.583235 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:15.584507 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:15.584993 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:15.585000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:15.585322 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:15.585782 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:15.585794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.586554 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.586557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:15.587420 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:15.587429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.587900 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.587922 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:15.588128 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:15.588725 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:15.588746 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:15.590379 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:15.592358 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:15.592364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:15.592377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:15.592633 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:15.592940 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:15.592944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:15.592946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:15.593087 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:15.593479 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:15.593486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:15.593723 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 06:00:15.593923 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 06:00:15.594221 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260808 06:00:15.596493 20882 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:15.596722 20882 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:15.597203 20882 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:15.598924 20685 ts_manager.cc:194] Registered new tserver with Master: 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
WARNING: no policy specified for oryx/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:00:15.600317 20685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:53365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
I20260808 06:00:15.617048 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:15.728015 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:15.728291 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:15.728372 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:15.732402 20887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:15.732481 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:15.732574 20887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:15.737813 20887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:15.738899 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:15.740021 20887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:15.749862 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.749838 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.749838 20893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:15.750376 20887 server_base.cc:1034] running on GCE node
I20260808 06:00:15.750779 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:15.751423 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:15.752609 20887 hybrid_clock.cc:648] HybridClock initialized: now 1786168815752593 us; error 24 us; skew 500 ppm
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:15.755519 20887 init.cc:377] Logged in from keytab as oryx/127.13.229.66@KRBTEST.COM (short username oryx)
I20260808 06:00:15.756456 20887 webserver.cc:460] Webserver started at http://127.13.229.66:41241/ using document root <none> and password file <none>
I20260808 06:00:15.756966 20887 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:15.757022 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:15.757189 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:15.759151 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "aab97520046a4864ae49fa62bebaf6c2"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "6a406963bd8f1b5662f3f80bbcb9175e"
server_key_iv: "83c2c979c8e46141b7a60ca38794d0cc"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.759670 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "aab97520046a4864ae49fa62bebaf6c2"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "6a406963bd8f1b5662f3f80bbcb9175e"
server_key_iv: "83c2c979c8e46141b7a60ca38794d0cc"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.763212 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:00:15.765532 20902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.766712 20887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:15.766831 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "aab97520046a4864ae49fa62bebaf6c2"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "6a406963bd8f1b5662f3f80bbcb9175e"
server_key_iv: "83c2c979c8e46141b7a60ca38794d0cc"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.766934 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:15.783236 20887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:15.786191 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:15.786409 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:15.787019 20887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:15.787865 20887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:15.788012 20887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.788089 20887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:15.788131 20887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.797263 20887 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:46725
I20260808 06:00:15.797291 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:46725 every 8 connection(s)
I20260808 06:00:15.798125 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:15.803660 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 20887
I20260808 06:00:15.803762 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:15.804114 14229 external_mini_cluster.cc:1428] Setting key 406a434997a5317c48d9d22196933d74
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:15.811923 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:15.799886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:41995 (local address 127.13.229.126:33945)
0808 06:00:15.800071 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:15.800075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:15.800891 (+ 816us) server_negotiation.cc:407] Connection header received
0808 06:00:15.802197 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:15.802200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:15.802248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:15.802325 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:15.803439 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.804124 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.805525 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.805749 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.805869 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:15.808444 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:15.808448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:15.808451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:15.808477 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:15.809859 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.810381 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.810385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.810387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.810473 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:15.810818 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:15.810823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:15.810824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:15.811015 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:15.811122 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:15.811599 (+ 477us) server_negotiation.cc:299] Negotiation successful
0808 06:00:15.811715 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:00:15.812676 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:15.800304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:41995)
0808 06:00:15.800735 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:15.800777 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:15.802000 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:15.802487 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:15.802494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:15.802827 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:15.803309 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:15.803322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.804287 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.804290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:15.805427 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:15.805440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:15.805909 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:15.805913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:15.806087 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:15.806692 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:15.806708 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:15.808278 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:15.809989 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:15.809995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:15.810009 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:15.810265 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:15.810572 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:15.810575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:15.810577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:15.810718 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:15.811121 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:15.811128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:15.811359 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 06:00:15.811552 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 06:00:15.811853 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260808 06:00:15.813825 21023 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:15.814085 21023 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:00:15.814680 21023 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:15.815809 20685 ts_manager.cc:194] Registered new tserver with Master: aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725)
I20260808 06:00:15.816401 20685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:41995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
I20260808 06:00:15.834421 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:15.945533 21028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:15.945910 21028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:15.945976 21028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:15.950019 21028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:15.950074 21028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:15.950132 21028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:15.955379 21028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:15.956382 21028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:15.957382 21028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:15.966857 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.966873 21034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:15.966853 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:15.967041 21028 server_base.cc:1034] running on GCE node
I20260808 06:00:15.967497 21028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:15.968153 21028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:15.969323 21028 hybrid_clock.cc:648] HybridClock initialized: now 1786168815969309 us; error 24 us; skew 500 ppm
Aug 08 06:00:15 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:15.972060 21028 init.cc:377] Logged in from keytab as oryx/127.13.229.67@KRBTEST.COM (short username oryx)
I20260808 06:00:15.972924 21028 webserver.cc:460] Webserver started at http://127.13.229.67:38945/ using document root <none> and password file <none>
I20260808 06:00:15.973474 21028 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:15.973564 21028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:15.973793 21028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:15.975721 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "611d3bccd07b40a6a700d4599f375ee7"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "7e5a0caf5acd9b9055453400c4cf5673"
server_key_iv: "32e0610426ba4dc42eeef061a4170d52"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.976352 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "611d3bccd07b40a6a700d4599f375ee7"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "7e5a0caf5acd9b9055453400c4cf5673"
server_key_iv: "32e0610426ba4dc42eeef061a4170d52"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.979934 21028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:15.982191 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:15.983363 21028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:15.983496 21028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "611d3bccd07b40a6a700d4599f375ee7"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "7e5a0caf5acd9b9055453400c4cf5673"
server_key_iv: "32e0610426ba4dc42eeef061a4170d52"
server_key_version: "encryptionkey@0"
I20260808 06:00:15.983603 21028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.027249 21028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.030764 21028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:16.030997 21028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.031649 21028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:16.032533 21028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:16.032605 21028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.032681 21028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:16.032727 21028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.041980 21028 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38289
I20260808 06:00:16.041999 21167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38289 every 8 connection(s)
I20260808 06:00:16.042958 21028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:16.051558 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21028
I20260808 06:00:16.051695 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:16.052063 14229 external_mini_cluster.cc:1428] Setting key 5470268570e7b1ba7f6f1e2aeee57c59
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:16.057536 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.044832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:44213 (local address 127.13.229.126:33945)
0808 06:00:16.044979 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.044982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.045796 (+ 814us) server_negotiation.cc:407] Connection header received
0808 06:00:16.047287 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.047291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.047376 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.047495 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.048753 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.049387 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.050520 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.050753 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.050876 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.053836 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.053843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.053851 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.053876 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.055439 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.055999 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.056003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.056005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.056091 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.056442 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.056446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.056447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.056628 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.056733 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.057243 (+ 510us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.057374 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:00:16.058283 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.045200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:44213)
0808 06:00:16.045652 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.045694 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.047078 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.047673 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.047680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.048104 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.048610 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.048623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.049512 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.049522 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.050392 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.050402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.050909 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.050913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.051086 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.051669 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.051692 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.053652 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.055564 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.055569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.055583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.055834 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.056188 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.056192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.056194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.056337 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.056748 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.056755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.056995 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.057184 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.057486 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260808 06:00:16.059336 21168 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:16.059584 21168 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:16.060134 21168 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:16.061151 20685 ts_manager.cc:194] Registered new tserver with Master: 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:16.061707 20685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:44213
I20260808 06:00:16.065917 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.229.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.125.keytab.
Entry for principal oryx/127.13.229.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.125.keytab.
I20260808 06:00:16.094328 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:35829
--webserver_interface=127.13.229.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.125.keytab
--principal=oryx/127.13.229.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33945,127.13.229.125:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:16.205531 21173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:16.205787 21173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:16.205864 21173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:16.209960 21173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:16.210016 21173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:16.210034 21173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:16.210075 21173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:16.210106 21173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:16.215209 21173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33945,127.13.229.125:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.125.keytab
--principal=oryx/127.13.229.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.229.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
W20260808 06:00:16.215415 21173 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.229.126:33945,127.13.229.125:35829'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 06:00:16.216339 21173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:16.217362 21173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:16.226122 21181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.226164 21179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.226178 21178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.226282 21173 server_base.cc:1034] running on GCE node
I20260808 06:00:16.226949 21173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:16.228058 21173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:16.229241 21173 hybrid_clock.cc:648] HybridClock initialized: now 1786168816229206 us; error 47 us; skew 500 ppm
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:16.232075 21173 init.cc:377] Logged in from keytab as oryx/127.13.229.125@KRBTEST.COM (short username oryx)
I20260808 06:00:16.232859 21173 webserver.cc:460] Webserver started at http://127.13.229.125:44921/ using document root <none> and password file <none>
I20260808 06:00:16.233407 21173 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.233472 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.233630 21173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:16.235755 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/instance:
uuid: "d33930902ee24abfb32c72e2cc38a77e"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "3a741406cdaf0164c1f29cc935a244e5"
server_key_iv: "ba8105f0febba5750bd14a55ab769c4e"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.236356 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal/instance:
uuid: "d33930902ee24abfb32c72e2cc38a77e"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "3a741406cdaf0164c1f29cc935a244e5"
server_key_iv: "ba8105f0febba5750bd14a55ab769c4e"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.239959 21173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:00:16.242215 21188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.243403 21173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:00:16.243551 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "d33930902ee24abfb32c72e2cc38a77e"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "3a741406cdaf0164c1f29cc935a244e5"
server_key_iv: "ba8105f0febba5750bd14a55ab769c4e"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.243654 21173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.255824 21173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.258550 21173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:16.258751 21173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.265686 21173 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.125:35829
I20260808 06:00:16.266009 21243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.125:35829 every 8 connection(s)
I20260808 06:00:16.266659 21173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
I20260808 06:00:16.269536 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:16.270714 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21173
I20260808 06:00:16.274091 21244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.125@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:16.285550 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.276047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.13.229.126:33945)
0808 06:00:16.276194 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.276199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.276502 (+ 303us) server_negotiation.cc:407] Connection header received
0808 06:00:16.276672 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.276675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.276732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.276817 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.277427 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.277969 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.278862 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.279276 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.279406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.281831 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.281834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.281836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.281858 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.283488 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.284179 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.284186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.284190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.284285 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.284554 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.284557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.284558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.284796 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.284905 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.285310 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.285435 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:00:16.286401 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.275972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44686)
0808 06:00:16.276360 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.276416 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.276534 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.276864 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.276870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.277174 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.277290 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.277295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.278106 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.278109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.278735 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.278744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.279422 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.279425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.279501 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.280075 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.280094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.281684 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.283612 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.283619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.283628 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.284087 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.284376 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.284379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.284381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.284460 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.284908 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.284914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.285066 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.285198 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.285490 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":101,"spinlock_wait_cycles":4864,"thread_start_us":173,"threads_started":1}
I20260808 06:00:16.286401 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.272289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.13.229.125:35829)
0808 06:00:16.272708 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.272727 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.272771 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:00:16.273601 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.273626 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.274044 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.274415 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.275154 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.276453 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.277444 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.277879 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.277986 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.281133 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.281141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.281153 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.281180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.283078 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.283618 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.283624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.283625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.283872 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.284220 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.284223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.284224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.284689 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.284868 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.285143 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.285435 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":1408,"thread_start_us":108,"threads_started":1}
I20260808 06:00:16.288358 21244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.125@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
I20260808 06:00:16.296543 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.288802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57948)
0808 06:00:16.288986 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.289000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.289085 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.289311 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.289314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.289470 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.289544 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.289549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.290584 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.290589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.291084 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.291090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.291534 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.291537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.291670 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.292182 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.292202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.293367 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.295081 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.295085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.295087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.295278 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.295549 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.295552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.295554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.295633 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.296037 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.296043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.296127 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.296231 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.296365 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":15}
I20260808 06:00:16.296548 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.288900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.13.229.125:35829)
0808 06:00:16.288988 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.288993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.289049 (+ 56us) server_negotiation.cc:407] Connection header received
0808 06:00:16.289152 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.289156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.289195 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.289301 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.289658 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.290476 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.291194 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.291422 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.291524 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.293493 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.293496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.293499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.293525 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.294957 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.295373 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.295378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.295379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.295453 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.295723 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.295727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.295728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.295936 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.296028 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.296277 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.296417 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":28}
I20260808 06:00:16.299832 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Bootstrap starting.
I20260808 06:00:16.302603 21244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:16.303421 21244 log.cc:824] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:16.305766 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: No bootstrap required, opened a new log
I20260808 06:00:16.308976 21244 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } }
I20260808 06:00:16.309158 21244 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:16.309191 21244 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33930902ee24abfb32c72e2cc38a77e, State: Initialized, Role: FOLLOWER
I20260808 06:00:16.309576 21244 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } }
I20260808 06:00:16.309995 21250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } } }
I20260808 06:00:16.310132 21250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:16.310705 21244 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: configured and running, proceeding with master startup.
W20260808 06:00:16.315603 21261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:00:16.315686 21261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:00:16.317565 21173 master_runner.cc:428] Detected that this master d33930902ee24abfb32c72e2cc38a77e is joining an existing cluster
I20260808 06:00:16.317631 21173 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.229.125:35829
I20260808 06:00:16.328655 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.320846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.13.229.125:35829)
0808 06:00:16.320950 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.320956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.320967 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:00:16.320999 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.321003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.321043 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.321125 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.321637 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.322285 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323106 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.323381 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323485 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.324879 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.324882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.324885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.324908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.326774 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.327287 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.327291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.327293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.327383 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.327683 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.327686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.327688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.327920 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.328018 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.328350 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.328498 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":44}
I20260808 06:00:16.328667 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.320468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57952)
0808 06:00:16.320805 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.320819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.320902 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.321163 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.321168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.321389 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.321516 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.321524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.322390 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.322393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.322984 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.322992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.323496 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.323645 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.324206 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.324227 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.324744 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.326902 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.326908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.326911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.327172 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.327492 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.327495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.327497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.327590 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.328002 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.328005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.328119 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.328281 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.328433 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":20,"thread_start_us":108,"threads_started":1}
I20260808 06:00:16.329952 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.320004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44698)
0808 06:00:16.320286 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.320303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.320418 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.320839 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.320842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.321064 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.321227 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.321233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.322207 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.322210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.323182 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.323194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.323979 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.324093 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.324545 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.324561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.325069 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.327981 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.327986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.327990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.328281 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.328597 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.328599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.328601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.328683 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.329490 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.329494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.329587 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.329685 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.329817 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260808 06:00:16.330058 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.320059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.13.229.126:33945)
0808 06:00:16.320203 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.320207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.320365 (+ 158us) server_negotiation.cc:407] Connection header received
0808 06:00:16.320555 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.320561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.320619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.320700 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.321339 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.322092 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323356 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.323709 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.323801 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.325240 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.325246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.325250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.325289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.327580 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.328392 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.328396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.328398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.328482 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.329205 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.329208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.329210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.329389 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.329497 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.329792 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.329904 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.126@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
I20260808 06:00:16.343652 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.334199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57956)
0808 06:00:16.334515 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.334562 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.334687 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.334973 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.334978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.335274 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.335379 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.335384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.336353 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.336355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.336966 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.336975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.337510 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.337512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.337593 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.338112 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.338124 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.339715 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.341806 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.341816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.341822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.342112 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.342418 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.342421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.342423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.342515 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.343004 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.343011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.343136 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.343286 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.343473 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260808 06:00:16.343691 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.334236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.13.229.125:35829)
0808 06:00:16.334337 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.334341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.334642 (+ 301us) server_negotiation.cc:407] Connection header received
0808 06:00:16.334816 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.334819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.334867 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.334953 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.335489 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.336253 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.337074 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.337403 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.337503 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.339874 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.339878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.339880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.339937 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.341688 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.342216 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.342222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.342226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.342315 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.342615 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.342621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.342624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.342882 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.342994 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.343369 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.343497 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:00:16.345187 20685 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.229.125:35829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e"
member_type: NON_VOTER
last_known_addr {
host: "127.13.229.125"
port: 35829
}
attrs {
promote: true
}
}
dest_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff"
cas_config_opid_index: -1
I20260808 06:00:16.345892 20685 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } }
I20260808 06:00:16.348148 21274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d33930902ee24abfb32c72e2cc38a77e (127.13.229.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } }
I20260808 06:00:16.348717 21216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:16.350238 21216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 06:00:16.350512 21274 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.229.125:35829
I20260808 06:00:16.350818 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } }
I20260808 06:00:16.351001 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.13.229.124@KRBTEST.COM; defaulting to no policy
I20260808 06:00:16.351143 21173 master.cc:537] Master@127.13.229.125:35829 shutting down...
I20260808 06:00:16.351305 21274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.124@KRBTEST.COM" created.
W20260808 06:00:16.354166 20666 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 06:00:16.355046 20666 consensus_peers.cc:487] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff -> Peer d33930902ee24abfb32c72e2cc38a77e (127.13.229.125:35829): Couldn't send request to peer d33930902ee24abfb32c72e2cc38a77e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 06:00:16.357795 21173 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:16.358103 21173 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:16.358233 21173 tablet_replica.cc:330] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: stopping tablet replica
I20260808 06:00:16.363334 21173 master.cc:552] Master@127.13.229.125:35829 shutdown complete.
I20260808 06:00:16.363390 21173 master_runner.cc:305] Clearing existing system tablet
I20260808 06:00:16.363673 21173 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.363730 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.365713 21173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.367594 21281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.367861 21173 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.368017 21173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.368062 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "d33930902ee24abfb32c72e2cc38a77e"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "3a741406cdaf0164c1f29cc935a244e5"
server_key_iv: "ba8105f0febba5750bd14a55ab769c4e"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.371001 21173 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.124.keytab.
Entry for principal oryx/127.13.229.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.124.keytab.
I20260808 06:00:16.374476 21173 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 06:00:16.374577 21173 log.cc:1196] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 06:00:16.374563 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:46803
--webserver_interface=127.13.229.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.124.keytab
--principal=oryx/127.13.229.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:00:16.374857 21173 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Deleting consensus metadata
I20260808 06:00:16.375058 21173 master_runner.cc:315] Copying system tablet from 127.13.229.126:33945
I20260808 06:00:16.376282 21173 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.229.126:33945
I20260808 06:00:16.385053 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.376946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.13.229.126:33945)
0808 06:00:16.377095 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.377102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.377386 (+ 284us) server_negotiation.cc:407] Connection header received
0808 06:00:16.377537 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.377541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.377598 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.377688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.378195 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.379021 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.380340 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.380566 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.380681 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.381822 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.381826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.381828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.381860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.383297 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.383742 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.383750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.383751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.383826 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.384176 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.384179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.384181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.384389 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.384503 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.384769 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.384883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:16.385062 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.376817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44708)
0808 06:00:16.377257 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.377275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.377418 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.377734 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.377736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.377962 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.378075 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.378082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.379137 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.379140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.380218 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.380228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.380686 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.380689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.380793 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.381363 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.381382 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.381678 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.383418 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.383423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.383425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.383633 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.383951 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.383957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.383961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.384051 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.384495 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.384498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.384602 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.384712 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.384864 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":94,"threads_started":1}
I20260808 06:00:16.385699 20705 tablet_copy_service.cc:140] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d33930902ee24abfb32c72e2cc38a77e ({username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:44708)
I20260808 06:00:16.385914 20705 tablet_copy_service.cc:161] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d33930902ee24abfb32c72e2cc38a77e at {username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:44708: session id = d33930902ee24abfb32c72e2cc38a77e-00000000000000000000000000000000
I20260808 06:00:16.388289 20705 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 06:00:16.390089 21173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 06:00:16.392346 21173 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: tablet copy: Starting download of 0 data blocks...
I20260808 06:00:16.392530 21173 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: tablet copy: Starting download of 1 WAL segments...
I20260808 06:00:16.394235 21173 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 06:00:16.396023 20705 tablet_copy_service.cc:342] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Request end of tablet copy session d33930902ee24abfb32c72e2cc38a77e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:44708
I20260808 06:00:16.396137 20705 tablet_copy_service.cc:433] P bb01fc67f68e440e95e9f6d0bf4cd8ff: ending tablet copy session d33930902ee24abfb32c72e2cc38a77e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d33930902ee24abfb32c72e2cc38a77e
I20260808 06:00:16.404361 21173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:16.405994 21292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.406384 21173 server_base.cc:1034] running on GCE node
W20260808 06:00:16.406643 21290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.407366 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.407601 21173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:16.407912 21173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:16.409065 21173 hybrid_clock.cc:648] HybridClock initialized: now 1786168816409041 us; error 38 us; skew 500 ppm
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:16.410748 21173 init.cc:377] Logged in from keytab as oryx/127.13.229.125@KRBTEST.COM (short username oryx)
I20260808 06:00:16.411183 21173 webserver.cc:460] Webserver started at http://127.13.229.125:39057/ using document root <none> and password file <none>
I20260808 06:00:16.411356 21173 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.411391 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.412804 21173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.413795 21299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.414053 21173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.414165 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "d33930902ee24abfb32c72e2cc38a77e"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "3a741406cdaf0164c1f29cc935a244e5"
server_key_iv: "ba8105f0febba5750bd14a55ab769c4e"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.414301 21173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.432585 21173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.434839 21173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.439833 21173 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.125:35829
I20260808 06:00:16.440088 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.125:35829 every 8 connection(s)
I20260808 06:00:16.440621 21173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
I20260808 06:00:16.442471 21354 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:00:16.443555 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Bootstrap starting.
I20260808 06:00:16.453994 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:00:16.454321 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Bootstrap complete.
I20260808 06:00:16.454885 21354 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } }
I20260808 06:00:16.455020 21354 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d33930902ee24abfb32c72e2cc38a77e, State: Initialized, Role: LEARNER
I20260808 06:00:16.455147 21354 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } }
I20260808 06:00:16.455495 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } }
I20260808 06:00:16.455603 21354 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:16.455686 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: LEARNER
I20260808 06:00:16.458997 21368 catalog_manager.cc:1259] Loaded cluster ID: ae1d44125a27488caafaddc63bca4f1b
I20260808 06:00:16.459059 21368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: loading cluster ID for follower catalog manager: success
I20260808 06:00:16.461213 21368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: acquiring CA information for follower catalog manager: success
I20260808 06:00:16.462606 21368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 06:00:16.506775 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:16.507041 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:16.507120 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:16.511049 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:16.511132 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:16.511171 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:16.511230 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:16.511268 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:16.519078 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.124.keytab
--principal=oryx/127.13.229.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.229.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:16.520562 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:16.521679 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:16.531356 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.531483 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.531512 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.531709 21286 server_base.cc:1034] running on GCE node
I20260808 06:00:16.532532 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:16.533752 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:16.534942 21286 hybrid_clock.cc:648] HybridClock initialized: now 1786168816534915 us; error 47 us; skew 500 ppm
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:16.538125 21286 init.cc:377] Logged in from keytab as oryx/127.13.229.124@KRBTEST.COM (short username oryx)
I20260808 06:00:16.539327 21286 webserver.cc:460] Webserver started at http://127.13.229.124:46409/ using document root <none> and password file <none>
I20260808 06:00:16.540163 21286 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.540266 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.540571 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:16.543701 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/instance:
uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "d5a45131326be9d09cb8e8d42f1f3207"
server_key_iv: "2be438e1dc79fbe28e90f30da7fed6ac"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.544523 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal/instance:
uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "d5a45131326be9d09cb8e8d42f1f3207"
server_key_iv: "2be438e1dc79fbe28e90f30da7fed6ac"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.549957 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 06:00:16.553390 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.555128 21286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 06:00:16.555271 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "d5a45131326be9d09cb8e8d42f1f3207"
server_key_iv: "2be438e1dc79fbe28e90f30da7fed6ac"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.555405 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.578958 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.582398 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:16.582590 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.590169 21286 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.124:46803
I20260808 06:00:16.590509 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.124:46803 every 8 connection(s)
I20260808 06:00:16.591156 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
I20260808 06:00:16.593911 21442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:16.595031 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21286
I20260808 06:00:16.599309 21442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:00:16.602772 20882 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.124@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:16.610082 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.596402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.13.229.124:46803)
0808 06:00:16.596783 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.596798 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.596831 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:00:16.597703 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.597722 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.598036 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.598399 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.599166 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.600288 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.601228 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.601943 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.602060 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.605155 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.605164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.605180 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.605210 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.607094 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.607546 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.607552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.607558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.607691 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.608028 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.608033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.608036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.608488 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.608681 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.609051 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.609353 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260808 06:00:16.611321 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.601157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.13.229.126:33945)
0808 06:00:16.601309 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.601314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.601486 (+ 172us) server_negotiation.cc:407] Connection header received
0808 06:00:16.601612 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.601615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.601666 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.601754 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.602560 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.603320 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.604773 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.604978 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.605088 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.607637 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.607641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.607644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.607670 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.609372 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.609831 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.609835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.609837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.609926 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.610376 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.610382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.610386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.610582 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.610703 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.611028 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.611147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 06:00:16.611334 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.600943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44716)
0808 06:00:16.601349 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.601396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.601513 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.601813 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.601818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.602220 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.602432 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.602440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.603462 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.603469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.604568 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.604581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.605136 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.605141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.605239 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.605933 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.605945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.607525 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.609484 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.609494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.609500 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.609718 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.610023 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.610026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.610028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.610110 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.610701 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.610707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.610822 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.610970 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.611148 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":179,"threads_started":1}
I20260808 06:00:16.612799 21442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.124@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
I20260808 06:00:16.620949 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.613298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57958)
0808 06:00:16.613415 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.613427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.613486 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.613932 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.613936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.614084 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.614168 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.614173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.615101 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.615106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.615614 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.615622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.616137 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.616141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.616241 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.616687 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.616698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.617868 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.619609 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.619611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.619612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.619793 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.620182 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.620185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.620186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.620260 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.620597 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.620599 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.620669 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.620746 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.620844 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 06:00:16.621197 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.613461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.13.229.125:35829)
0808 06:00:16.613709 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.613713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.613724 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:00:16.613776 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.613778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.613815 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.613916 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.614291 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.614988 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.615733 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.616023 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.616135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.618015 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.618018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.618020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.618043 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.619510 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.619948 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.619954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.619959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.620096 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.620349 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.620352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.620353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.620517 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.620594 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.620843 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.620958 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260808 06:00:16.621855 21442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.124@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
I20260808 06:00:16.629091 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.622269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39026)
0808 06:00:16.622382 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.622397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.622464 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.622678 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.622680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.622815 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.622880 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.622885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.624018 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.624021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.624493 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.624500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.624898 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.624900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.624955 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.625302 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.625312 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.626359 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.627824 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.627826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.627828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.628063 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.628369 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.628371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.628373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.628444 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.628733 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.628735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.628822 (+ 87us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.628899 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.628992 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 06:00:16.629170 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.622361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.13.229.124:46803)
0808 06:00:16.622474 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.622479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.622495 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:16.622535 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.622538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.622574 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.622658 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.622978 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.623842 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.624588 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.624805 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.624882 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.626495 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.626498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.626500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.626521 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.627695 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.628191 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.628194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.628195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.628277 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.628502 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.628504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.628505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.628657 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.628759 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.628933 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.629048 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":40}
I20260808 06:00:16.632004 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Bootstrap starting.
I20260808 06:00:16.634250 21442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:16.634974 21442 log.cc:824] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:16.636837 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: No bootstrap required, opened a new log
I20260808 06:00:16.639683 21442 raft_consensus.cc:348] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } }
I20260808 06:00:16.639899 21442 raft_consensus.cc:374] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:16.640000 21442 raft_consensus.cc:729] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f192ba9e88d64933b0ad1b7fb31c4cb8, State: Initialized, Role: FOLLOWER
I20260808 06:00:16.640450 21442 consensus_queue.cc:260] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } }
I20260808 06:00:16.640897 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } } }
I20260808 06:00:16.641008 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:16.641613 21442 sys_catalog.cc:564] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 06:00:16.646342 21460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:00:16.646440 21460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:00:16.649739 21286 master_runner.cc:428] Detected that this master f192ba9e88d64933b0ad1b7fb31c4cb8 is joining an existing cluster
I20260808 06:00:16.649801 21286 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.229.124:46803
I20260808 06:00:16.661900 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.13.229.126:33945)
0808 06:00:16.652292 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.652296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.652958 (+ 662us) server_negotiation.cc:407] Connection header received
0808 06:00:16.653125 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.653128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.653185 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.653283 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.653898 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.654737 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656086 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.656426 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656505 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.657980 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.657984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.657987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.658013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.659803 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.660496 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.660501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.660503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.660607 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.660978 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.660982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.660984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.661181 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.661286 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.661578 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.661697 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:00:16.661962 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44724)
0808 06:00:16.652855 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.652870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.652977 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.653309 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.653314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.653609 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.653774 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.653781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.654924 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.654927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.655949 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.655961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.656674 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.656801 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.657376 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.657392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.657826 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.659951 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.659954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.659957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.660356 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.660726 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.660776 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.660781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.660879 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.661263 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.661266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.661360 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.661500 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.661630 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":19,"spinlock_wait_cycles":16640,"thread_start_us":133,"threads_started":1}
I20260808 06:00:16.662220 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.13.229.124:46803)
0808 06:00:16.652591 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.652595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.653021 (+ 426us) server_negotiation.cc:407] Connection header received
0808 06:00:16.653508 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.653512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.653553 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.653599 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.654172 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.654902 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656965 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.657298 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.657386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.658615 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.658618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.658620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.658651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.660292 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.660822 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.660825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.660827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.660918 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.661213 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.661220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.661224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.661491 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.661589 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.661929 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.662037 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":31}
I20260808 06:00:16.663120 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39032)
0808 06:00:16.652947 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.652959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.653049 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.653760 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.653763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.653976 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.654065 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.654071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.655670 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.655673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.656834 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.656844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.657467 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.657469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.657565 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.658042 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.658056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.658460 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.660443 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.660446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.660448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.660715 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.661019 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.661021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.661026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.661115 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.661622 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.661625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.661724 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.662809 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.662951 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":6,"thread_start_us":106,"threads_started":1}
I20260808 06:00:16.663679 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20748
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168815, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.126@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
I20260808 06:00:16.679188 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:41229
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:33945
--webserver_port=38959
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:00:16.680925 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.13.229.125:35829)
0808 06:00:16.652671 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.652675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.653322 (+ 647us) server_negotiation.cc:407] Connection header received
0808 06:00:16.653475 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.653479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.653543 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.653620 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.654614 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.655290 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656532 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.656881 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.656979 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.677176 (+ 20197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.677180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.677183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.677219 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.678887 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.679441 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.679445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.679447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.679531 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.679872 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.679878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.679880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.680141 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.680213 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.680589 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.680728 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 06:00:16.681499 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.652509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57964)
0808 06:00:16.653223 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.653259 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.653381 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.654201 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.654204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.654382 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.654484 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.654490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.655401 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.655403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.656403 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.656414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.657130 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.657133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.657272 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.676609 (+ 19337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.676629 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.677008 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.679030 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.679035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.679039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.679311 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.679647 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.679653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.679658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.679749 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.680292 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.680296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.680393 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.681137 (+ 744us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.681300 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":102,"thread_start_us":104,"threads_started":1}
I20260808 06:00:16.682180 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.666065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39048)
0808 06:00:16.666217 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.666237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.666330 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.666643 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.666649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.666885 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.667029 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.667042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.671793 (+ 4751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.671800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.672813 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.672828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.673425 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.673431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.673561 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.674085 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.674104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.675819 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.680085 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.680088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.680090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.680336 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.680644 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.680647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.680649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.680737 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.681440 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.681443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.681772 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.681892 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.682037 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 06:00:16.682821 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.666239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.13.229.124:46803)
0808 06:00:16.666345 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.666349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.666362 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:16.666475 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.666478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.666525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.666603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.667162 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.667930 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.672950 (+ 5020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.673287 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.673390 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.676006 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.676010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.676012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.676038 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.679971 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.680454 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.680459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.680464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.680544 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.680842 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.680845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.680847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.681293 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.681393 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.682433 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.682539 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":49}
I20260808 06:00:16.683665 20685 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.229.124:46803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
member_type: NON_VOTER
last_known_addr {
host: "127.13.229.124"
port: 46803
}
attrs {
promote: true
}
}
dest_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff"
cas_config_opid_index: 5
I20260808 06:00:16.684470 20685 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } }
I20260808 06:00:16.685977 21274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.687844 21274 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.229.124:46803
I20260808 06:00:16.688081 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.688212 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:16.688308 21286 master.cc:537] Master@127.13.229.124:46803 shutting down...
I20260808 06:00:16.689671 21414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:16.692826 21414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 06:00:16.693492 21275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:00:16.695611 20667 consensus_peers.cc:487] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff -> Peer f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124:46803): Couldn't send request to peer f192ba9e88d64933b0ad1b7fb31c4cb8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 06:00:16.696427 21286 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:16.696702 21286 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:16.696794 21286 tablet_replica.cc:330] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: stopping tablet replica
I20260808 06:00:16.701295 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.692121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.13.229.125:35829)
0808 06:00:16.692358 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.692374 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.694498 (+ 2124us) server_negotiation.cc:407] Connection header received
0808 06:00:16.695075 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.695078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.695135 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.695198 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:16.695582 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.696393 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.699586 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.700257 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.700370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.700513 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.701083 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.701172 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260808 06:00:16.701673 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.685821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:57972)
0808 06:00:16.694421 (+ 8600us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.694438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.694942 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.695344 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.695347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.695372 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:16.695464 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.695470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.696536 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.696539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.699451 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.699462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.700753 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.700760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.700874 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.700941 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.701484 (+ 543us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.701561 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8498}
I20260808 06:00:16.702210 21327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 LEARNER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 06:00:16.702379 21286 master.cc:552] Master@127.13.229.124:46803 shutdown complete.
I20260808 06:00:16.702455 21286 master_runner.cc:305] Clearing existing system tablet
I20260808 06:00:16.702612 21275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 06:00:16.703069 21286 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.703130 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.704367 21286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 06:00:16.704561 21327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.705189 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.705361 21286 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.705457 21286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 06:00:16.705487 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "d5a45131326be9d09cb8e8d42f1f3207"
server_key_iv: "2be438e1dc79fbe28e90f30da7fed6ac"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.706825 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb01fc67f68e440e95e9f6d0bf4cd8ff. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.706945 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: LEARNER
I20260808 06:00:16.707046 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: NON_VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: true } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.707123 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: LEARNER
I20260808 06:00:16.707664 21286 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:16.709959 21286 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 06:00:16.710049 21286 log.cc:1196] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 06:00:16.710300 21286 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Deleting consensus metadata
I20260808 06:00:16.710462 21286 master_runner.cc:315] Copying system tablet from 127.13.229.126:33945
I20260808 06:00:16.711539 21286 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: tablet copy: Beginning tablet copy session from remote peer at address 127.13.229.126:33945
I20260808 06:00:16.713418 21275 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: attempting to promote NON_VOTER d33930902ee24abfb32c72e2cc38a77e to VOTER
I20260808 06:00:16.713889 21275 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } }
I20260808 06:00:16.715071 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.714794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39060)
0808 06:00:16.714908 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.714999 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 06:00:16.715441 21327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 LEARNER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 06:00:16.715871 21272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 06:00:16.716472 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.716229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39062)
0808 06:00:16.716356 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.716413 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 06:00:16.717103 20667 consensus_peers.cc:487] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff -> Peer f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124:46803): Couldn't send request to peer f192ba9e88d64933b0ad1b7fb31c4cb8. Status: Network error: Client connection negotiation failed: client connection to 127.13.229.124:46803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 06:00:16.717691 21274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d33930902ee24abfb32c72e2cc38a77e (127.13.229.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.718123 21327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d33930902ee24abfb32c72e2cc38a77e (127.13.229.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.719038 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.719135 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:16.719256 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.719326 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:16.719507 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.719614 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:16.736565 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.712066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.13.229.126:33945)
0808 06:00:16.712171 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.712175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.728211 (+ 16036us) server_negotiation.cc:407] Connection header received
0808 06:00:16.728334 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.728337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.728387 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.728472 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.729006 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.729808 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.731004 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.731350 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.731455 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.732835 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.732839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.732842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.732870 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.734601 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.735163 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.735166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.735168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.735287 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.735637 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.735640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.735642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.735861 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.735984 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.736299 (+ 315us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.736413 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:00:16.736586 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.712033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:44736)
0808 06:00:16.728099 (+ 16066us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.728115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.728236 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.728515 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.728520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.728748 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.728882 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.728889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.729935 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.729939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.730874 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.730886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.731475 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.731479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.731577 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.732238 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.732253 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.732684 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.734730 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.734736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.734739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.735043 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.735404 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.735409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.735413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.735526 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.735993 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.735998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.736104 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.736219 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.736359 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15983,"thread_start_us":84,"threads_started":1}
I20260808 06:00:16.737413 20705 tablet_copy_service.cc:140] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f192ba9e88d64933b0ad1b7fb31c4cb8 ({username='oryx', principal='oryx/127.13.229.124@KRBTEST.COM'} at 127.0.0.1:44736)
I20260808 06:00:16.737514 20705 tablet_copy_service.cc:161] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f192ba9e88d64933b0ad1b7fb31c4cb8 at {username='oryx', principal='oryx/127.13.229.124@KRBTEST.COM'} at 127.0.0.1:44736: session id = f192ba9e88d64933b0ad1b7fb31c4cb8-00000000000000000000000000000000
I20260808 06:00:16.738577 20705 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 06:00:16.739856 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 06:00:16.742307 21286 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: tablet copy: Starting download of 0 data blocks...
I20260808 06:00:16.742506 21286 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: tablet copy: Starting download of 1 WAL segments...
I20260808 06:00:16.743983 21286 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 06:00:16.745853 20705 tablet_copy_service.cc:342] P bb01fc67f68e440e95e9f6d0bf4cd8ff: Request end of tablet copy session f192ba9e88d64933b0ad1b7fb31c4cb8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.229.124@KRBTEST.COM'} at 127.0.0.1:44736
I20260808 06:00:16.745936 20705 tablet_copy_service.cc:433] P bb01fc67f68e440e95e9f6d0bf4cd8ff: ending tablet copy session f192ba9e88d64933b0ad1b7fb31c4cb8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f192ba9e88d64933b0ad1b7fb31c4cb8
I20260808 06:00:16.753572 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:16.755232 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.756790 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.760735 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.761020 21286 server_base.cc:1034] running on GCE node
I20260808 06:00:16.761291 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:16.761585 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:16.762746 21286 hybrid_clock.cc:648] HybridClock initialized: now 1786168816762730 us; error 30 us; skew 500 ppm
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:16.764434 21286 init.cc:377] Logged in from keytab as oryx/127.13.229.124@KRBTEST.COM (short username oryx)
I20260808 06:00:16.764883 21286 webserver.cc:460] Webserver started at http://127.13.229.124:35933/ using document root <none> and password file <none>
I20260808 06:00:16.765069 21286 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.765110 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.766585 21286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.767701 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.768010 21286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:16.768085 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
format_stamp: "Formatted at 2026-08-08 06:00:16 on dist-test-slave-zkpd"
server_key: "d5a45131326be9d09cb8e8d42f1f3207"
server_key_iv: "2be438e1dc79fbe28e90f30da7fed6ac"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.768162 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.781263 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.783782 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.789711 21286 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.124:46803
I20260808 06:00:16.790378 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
I20260808 06:00:16.790947 21544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.124:46803 every 8 connection(s)
I20260808 06:00:16.792416 21545 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:00:16.793537 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Bootstrap starting.
I20260808 06:00:16.806538 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:00:16.806810 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: Bootstrap complete.
I20260808 06:00:16.807348 21545 raft_consensus.cc:348] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } }
I20260808 06:00:16.807487 21545 raft_consensus.cc:729] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f192ba9e88d64933b0ad1b7fb31c4cb8, State: Initialized, Role: LEARNER
I20260808 06:00:16.807590 21545 consensus_queue.cc:260] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } }
I20260808 06:00:16.808056 21545 sys_catalog.cc:564] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:16.808229 21548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:16.808462 21548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:00:16.812615 21559 catalog_manager.cc:1259] Loaded cluster ID: ae1d44125a27488caafaddc63bca4f1b
I20260808 06:00:16.812676 21559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: loading cluster ID for follower catalog manager: success
I20260808 06:00:16.814894 21559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: acquiring CA information for follower catalog manager: success
I20260808 06:00:16.816286 21559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:00:16.818645 21023 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
W20260808 06:00:16.825080 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:16.825321 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:16.825404 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:16.829504 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:16.829576 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:16.829685 21469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:16.834839 21469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:41229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=38959
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:16.835824 21469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:16.836932 21469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:16.846338 21568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.846316 21565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:16.846305 21566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:16.846671 21469 server_base.cc:1034] running on GCE node
I20260808 06:00:16.847102 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:16.847775 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:16.848929 21469 hybrid_clock.cc:648] HybridClock initialized: now 1786168816848900 us; error 43 us; skew 500 ppm
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:16.851543 21469 init.cc:377] Logged in from keytab as oryx/127.13.229.65@KRBTEST.COM (short username oryx)
I20260808 06:00:16.852411 21469 webserver.cc:460] Webserver started at http://127.13.229.65:38959/ using document root <none> and password file <none>
I20260808 06:00:16.852942 21469 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:16.853044 21469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:16.856869 21469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:16.858819 21577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.860051 21469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:16.860142 21469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "73778dd88e2a417e86297b0a56f16ddc"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "44087f2c09ca6e5f83f016346f3f8349"
server_key_iv: "cd02838198fbf17050fd4c030856d4ff"
server_key_version: "encryptionkey@0"
I20260808 06:00:16.860641 21469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:16.881924 21469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:16.884573 21469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:16.884719 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:16.885222 21469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:16.886003 21469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:16.886050 21469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.886088 21469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:16.886104 21469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:16.897383 21469 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:41229
I20260808 06:00:16.897419 21692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:41229 every 8 connection(s)
I20260808 06:00:16.898671 21469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:16.907881 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21469
I20260808 06:00:16.908128 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20887
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
Aug 08 06:00:16 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168816, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
I20260808 06:00:16.921352 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.902103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:58887 (local address 127.13.229.125:35829)
0808 06:00:16.902247 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.902252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.902510 (+ 258us) server_negotiation.cc:407] Connection header received
0808 06:00:16.903937 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.903942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.904006 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.904061 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.906121 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.906948 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.912874 (+ 5926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.913119 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.913237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.916625 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.916629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.916632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.916661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.918557 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919345 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.919349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.919350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.919436 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919920 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.919924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.919925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.920155 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.920229 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.921009 (+ 780us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.921136 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 06:00:16.921625 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.900680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:43759 (local address 127.13.229.126:33945)
0808 06:00:16.900860 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.900865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.902458 (+ 1593us) server_negotiation.cc:407] Connection header received
0808 06:00:16.903919 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.903926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.903996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.904056 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.906121 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.907011 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.912079 (+ 5068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.912502 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.912599 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.916625 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.916629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.916632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.916661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.919185 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919693 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.919696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.919698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.919777 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.920114 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.920117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.920119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.920365 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.920432 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.921009 (+ 577us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.921136 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 06:00:16.922556 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:46725
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:33945
--webserver_port=41241
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:00:16.923517 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.901950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.13.229.65:58887)
0808 06:00:16.902432 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.902444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.903669 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.904392 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.904401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.905119 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.905942 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.905964 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.907137 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.907140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.912734 (+ 5594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.912745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.913238 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.913241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.913337 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.913924 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.913937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.916425 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.918722 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.918728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.918744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.919231 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.919592 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.919597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.919785 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.920387 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.920397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.920720 (+ 323us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.921987 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.922367 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":214,"spinlock_wait_cycles":123520,"thread_start_us":93,"threads_started":1}
I20260808 06:00:16.923499 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.901094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:43759)
0808 06:00:16.902295 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.902350 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.903648 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.904393 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.904403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.905119 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.905942 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.905964 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.907161 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.907167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.911929 (+ 4762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.911944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.912823 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.912829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.913097 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.913848 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.913875 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.916425 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.919333 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.919337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.919585 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.919914 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.919917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.919919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.920005 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.920499 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.920502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.920720 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.921777 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.922417 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":237,"spinlock_wait_cycles":73984,"thread_start_us":128,"threads_started":1}
I20260808 06:00:16.925050 21696 heartbeater.cc:347] Connected to a master server at 127.13.229.125:35829
I20260808 06:00:16.925272 21695 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:16.925313 21696 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:16.925362 21695 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:16.925791 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.901094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.13.229.65:44875)
0808 06:00:16.902325 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:16.902361 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:16.903669 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:16.904392 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:16.904401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:16.905119 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:16.905943 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.905964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.907331 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.907335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.917191 (+ 9856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:16.917205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.918128 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.918134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:16.918269 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.918709 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:16.918728 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:16.920606 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:16.922898 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.922904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.922908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.924305 (+ 1397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.924651 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:16.924655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:16.924657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:16.924756 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:16.925283 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:16.925289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:16.925389 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 06:00:16.925487 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 06:00:16.925618 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":555,"spinlock_wait_cycles":60672,"thread_start_us":90,"threads_started":1}
I20260808 06:00:16.925917 21695 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:16.926292 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:16.901319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:44875 (local address 127.13.229.124:46803)
0808 06:00:16.901962 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:16.901977 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:16.902798 (+ 821us) server_negotiation.cc:407] Connection header received
0808 06:00:16.904004 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:16.904007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:16.904063 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:16.904138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:16.906121 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.907193 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.917344 (+ 10151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:16.917859 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:16.917976 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:16.920775 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:16.920782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:16.920788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:16.920822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:16.922770 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.924442 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.924445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.924447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.924541 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:16.924880 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:16.924884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:16.924886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:16.925129 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:16.925226 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:16.925995 (+ 769us) server_negotiation.cc:299] Negotiation successful
0808 06:00:16.926128 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":104,"threads_started":1}
I20260808 06:00:16.926944 21694 heartbeater.cc:347] Connected to a master server at 127.13.229.124:46803
I20260808 06:00:16.927029 21694 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:16.927201 21694 heartbeater.cc:511] Master 127.13.229.124:46803 requested a full tablet report, sending...
I20260808 06:00:16.927330 20685 ts_manager.cc:194] Re-registered known tserver with Master: 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
I20260808 06:00:16.928056 20685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:43759
I20260808 06:00:16.925921 21696 heartbeater.cc:511] Master 127.13.229.125:35829 requested a full tablet report, sending...
I20260808 06:00:16.928923 21508 ts_manager.cc:194] Registered new tserver with Master: 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
I20260808 06:00:16.929680 21317 ts_manager.cc:194] Registered new tserver with Master: 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
W20260808 06:00:17.038353 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:17.038597 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:17.038640 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:17.042928 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:17.042994 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:17.043074 21702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:17.049099 21702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:46725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=41241
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:17.050066 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:17.050952 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:17.060439 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:17.060451 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:17.060468 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:17.060873 21702 server_base.cc:1034] running on GCE node
I20260808 06:00:17.061260 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:17.061884 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:17.063043 21702 hybrid_clock.cc:648] HybridClock initialized: now 1786168817063014 us; error 43 us; skew 500 ppm
I20260808 06:00:17.063611 21168 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:17.066185 21702 init.cc:377] Logged in from keytab as oryx/127.13.229.66@KRBTEST.COM (short username oryx)
I20260808 06:00:17.067035 21702 webserver.cc:460] Webserver started at http://127.13.229.66:41241/ using document root <none> and password file <none>
I20260808 06:00:17.067536 21702 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:17.067603 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:17.071410 21702 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:17.073441 21719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.074635 21702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:17.074756 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "aab97520046a4864ae49fa62bebaf6c2"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "6a406963bd8f1b5662f3f80bbcb9175e"
server_key_iv: "83c2c979c8e46141b7a60ca38794d0cc"
server_key_version: "encryptionkey@0"
I20260808 06:00:17.075269 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:17.107481 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:17.110909 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:17.111150 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:17.111755 21702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:17.112637 21702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:17.112684 21702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.112752 21702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:17.112782 21702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.122247 21702 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:46725
I20260808 06:00:17.122282 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:46725 every 8 connection(s)
I20260808 06:00:17.123245 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:17.130650 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21702
I20260808 06:00:17.130844 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21028
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
I20260808 06:00:17.140394 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.125066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:33969 (local address 127.13.229.126:33945)
0808 06:00:17.125633 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.125638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.126128 (+ 490us) server_negotiation.cc:407] Connection header received
0808 06:00:17.127384 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.127389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.127443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.127501 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.128956 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.129839 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131332 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.131680 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131751 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.135469 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.135477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.135484 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.135523 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.137686 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138433 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.138437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.138439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.138528 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138984 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.138988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.138990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.139207 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.139281 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.140098 (+ 817us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.140220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260808 06:00:17.140394 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.126680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:35185 (local address 127.13.229.125:35829)
0808 06:00:17.126803 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.126806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.126857 (+ 51us) server_negotiation.cc:407] Connection header received
0808 06:00:17.127631 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.127635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.127685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.127822 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.128986 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.129506 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131557 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.131880 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.135729 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.135733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.135736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.135763 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.137885 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138437 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.138440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.138442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.138528 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138984 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.138988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.138990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.139217 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.139281 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.140098 (+ 817us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.140220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 06:00:17.141916 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.125059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:58633 (local address 127.13.229.124:46803)
0808 06:00:17.125188 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.125192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.126133 (+ 941us) server_negotiation.cc:407] Connection header received
0808 06:00:17.127346 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.127352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.127416 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.127501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.128994 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.129823 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131278 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.131780 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131890 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.136213 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.136221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.136227 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.136276 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.138983 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.139618 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.139625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.139630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.139737 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.140163 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.140170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.140175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.140480 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.140572 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.141502 (+ 930us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.141674 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:00:17.142031 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.125483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:33969)
0808 06:00:17.125980 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.126033 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.127132 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.127677 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.127679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.128184 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.128811 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.128829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.130240 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.130244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.131189 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.131214 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.131966 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.131971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.132161 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.133162 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.133180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.135232 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.137833 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.137842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.137857 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.138309 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.138648 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.138652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.138870 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.139371 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.139378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.139775 (+ 397us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.140558 (+ 783us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.140979 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":184,"spinlock_wait_cycles":41856,"thread_start_us":154,"threads_started":1}
I20260808 06:00:17.142386 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.126536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.13.229.66:35185)
0808 06:00:17.126784 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.126798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.127130 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.127831 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.127835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.128221 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.128822 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.128832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.129665 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.129668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.131428 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.131436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.132140 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.132144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.132309 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.132887 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.132910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.135589 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.138003 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.138008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.138309 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.138648 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.138650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.138652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.138863 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.139379 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.139381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.139775 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.140592 (+ 817us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.141027 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":842,"spinlock_wait_cycles":81920,"thread_start_us":83,"threads_started":1}
I20260808 06:00:17.143262 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.125483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.13.229.66:58633)
0808 06:00:17.125980 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.126033 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.127130 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.127649 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.127656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.128222 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.128825 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.128832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.129947 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.129950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.131103 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.131118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.132110 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.132117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.132409 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.133043 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.133066 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.135998 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.139137 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.139144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.139149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.139480 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.139880 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.139886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.139913 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.140030 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.141092 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.141098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.141232 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.142750 (+ 1518us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.142937 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":219,"spinlock_wait_cycles":9728,"thread_start_us":142,"threads_started":1}
I20260808 06:00:17.146328 21838 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:17.146279 21839 heartbeater.cc:347] Connected to a master server at 127.13.229.125:35829
I20260808 06:00:17.146605 21838 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.146610 21839 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.147194 21839 heartbeater.cc:511] Master 127.13.229.125:35829 requested a full tablet report, sending...
I20260808 06:00:17.147193 21838 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:17.148296 21837 heartbeater.cc:347] Connected to a master server at 127.13.229.124:46803
I20260808 06:00:17.148394 21837 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.148568 21837 heartbeater.cc:511] Master 127.13.229.124:46803 requested a full tablet report, sending...
I20260808 06:00:17.148576 21317 ts_manager.cc:194] Registered new tserver with Master: aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725)
I20260808 06:00:17.148726 20685 ts_manager.cc:194] Re-registered known tserver with Master: aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725)
I20260808 06:00:17.149312 21508 ts_manager.cc:194] Registered new tserver with Master: aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725)
I20260808 06:00:17.149317 20685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:33969
I20260808 06:00:17.151973 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:38289
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:33945
--webserver_port=38945
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:17.265517 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:17.265820 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:17.265918 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:17.270457 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:17.270516 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:17.270571 21844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:17.275625 21844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:38289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=38945
--tserver_master_addrs=127.13.229.126:33945,127.13.229.125:35829,127.13.229.124:46803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:17.276625 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:17.277552 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:17.287292 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:17.287317 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:17.287321 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:17.287528 21844 server_base.cc:1034] running on GCE node
I20260808 06:00:17.288040 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:17.288681 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:17.289870 21844 hybrid_clock.cc:648] HybridClock initialized: now 1786168817289853 us; error 26 us; skew 500 ppm
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:17.292565 21844 init.cc:377] Logged in from keytab as oryx/127.13.229.67@KRBTEST.COM (short username oryx)
I20260808 06:00:17.293368 21844 webserver.cc:460] Webserver started at http://127.13.229.67:38945/ using document root <none> and password file <none>
I20260808 06:00:17.294013 21844 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:17.294101 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:17.298609 21844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:17.300983 21860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.302459 21844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:17.302577 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "611d3bccd07b40a6a700d4599f375ee7"
format_stamp: "Formatted at 2026-08-08 06:00:15 on dist-test-slave-zkpd"
server_key: "7e5a0caf5acd9b9055453400c4cf5673"
server_key_iv: "32e0610426ba4dc42eeef061a4170d52"
server_key_version: "encryptionkey@0"
I20260808 06:00:17.303226 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:17.311609 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.307109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39076)
0808 06:00:17.307357 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.307375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.307525 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.307864 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.307868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.307924 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.308055 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.308064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.309243 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.309246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.310308 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.310321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.311225 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.311242 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.311321 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.311346 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.311419 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.311488 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260808 06:00:17.311632 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.307226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.13.229.124:46803)
0808 06:00:17.307357 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.307363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.307461 (+ 98us) server_negotiation.cc:407] Connection header received
0808 06:00:17.307633 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.307636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.307725 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.307836 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.308242 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.309111 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.310439 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.311081 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.311195 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.311317 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.311448 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.311513 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:17.312410 21518 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 06:00:17.312959 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb01fc67f68e440e95e9f6d0bf4cd8ff. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:17.313124 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: LEARNER
I20260808 06:00:17.317203 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:17.320305 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:17.320509 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:17.321051 21844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:17.321954 21844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:17.322019 21844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.322085 21844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:17.322132 21844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:17.331418 21844 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38289
I20260808 06:00:17.331450 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38289 every 8 connection(s)
I20260808 06:00:17.332403 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:17.339300 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 21844
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.124@KRBTEST.COM
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.125@KRBTEST.COM
I20260808 06:00:17.350821 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.334129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54007 (local address 127.13.229.124:46803)
0808 06:00:17.334314 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.334321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.335308 (+ 987us) server_negotiation.cc:407] Connection header received
0808 06:00:17.336583 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.336587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.336641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.336706 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.337935 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.338469 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340241 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.340604 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340699 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.345127 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.345131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.345134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.345160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.346932 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.348849 (+ 1917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.348854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.348856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.348946 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.349520 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.349524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.349527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.349763 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.349864 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.350517 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.350643 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:00:17.351848 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.341232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.13.229.126:33945)
0808 06:00:17.341487 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.341493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.341740 (+ 247us) server_negotiation.cc:407] Connection header received
0808 06:00:17.341846 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.341850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.341907 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.341999 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.342729 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.343464 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.344472 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.344825 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.344947 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.346596 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.346600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.346603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.346631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.349582 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.350103 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.350106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.350108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.350200 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.350565 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.350568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.350570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.350767 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.351497 (+ 730us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.351574 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.351699 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260808 06:00:17.352361 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.334528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.13.229.67:54007)
0808 06:00:17.335145 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.335205 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.336279 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.336828 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.336836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.337303 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.337805 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.337812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.338613 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.338618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.340079 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.340090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.340872 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.341229 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.342188 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.342212 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.344981 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.347062 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.347070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.347088 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.348698 (+ 1610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.349204 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.349207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.349210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.349393 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.349930 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.349936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.350227 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.350985 (+ 758us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.351310 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":304,"spinlock_wait_cycles":19456,"thread_start_us":153,"threads_started":1}
I20260808 06:00:17.353724 21985 heartbeater.cc:347] Connected to a master server at 127.13.229.124:46803
I20260808 06:00:17.354002 21985 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.354197 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.341956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.13.229.125:35829)
0808 06:00:17.343101 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.343105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.343117 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:00:17.343151 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.343154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.343194 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.343278 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.344194 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.344871 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.346039 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.346435 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.346909 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.348217 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.348221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.348223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.348249 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.351219 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.351711 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.351714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.351716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.351796 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.352239 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.352241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.352243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.352569 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.353583 (+ 1014us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.353890 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.354011 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":75,"threads_started":1}
I20260808 06:00:17.354568 21985 heartbeater.cc:511] Master 127.13.229.124:46803 requested a full tablet report, sending...
Aug 08 06:00:17 dist-test-slave-zkpd krb5kdc[20634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168817, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:17.355880 21508 ts_manager.cc:194] Registered new tserver with Master: 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:17.356387 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.334728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54779 (local address 127.13.229.125:35829)
0808 06:00:17.334883 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.334889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.335305 (+ 416us) server_negotiation.cc:407] Connection header received
0808 06:00:17.336511 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.336514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.336573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.336627 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.337953 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.338712 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340256 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.340621 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.347580 (+ 6884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.347586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.347588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.347616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.349811 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.353069 (+ 3258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.353073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.353075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.353169 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.353535 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.353539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.353540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.353801 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.353884 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.356123 (+ 2239us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.356231 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 06:00:17.356619 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.334528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.13.229.67:54779)
0808 06:00:17.335145 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.335205 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.336274 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.336828 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.336836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.337309 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.337789 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.337799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.339302 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.339324 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.340126 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.340135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.340872 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.340876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.341231 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.342188 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.342225 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.347456 (+ 5231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.352670 (+ 5214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.352673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.352675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.352936 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.353302 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.353305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.353307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.353393 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.353947 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.353949 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.354034 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.356044 (+ 2010us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.356228 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":233,"spinlock_wait_cycles":80000,"thread_start_us":131,"threads_started":1}
I20260808 06:00:17.354880 20685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 06:00:17.359620 21987 heartbeater.cc:347] Connected to a master server at 127.13.229.125:35829
W20260808 06:00:17.359637 20685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:17.359699 21987 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.359931 21987 heartbeater.cc:511] Master 127.13.229.125:35829 requested a full tablet report, sending...
I20260808 06:00:17.360596 21317 ts_manager.cc:194] Registered new tserver with Master: 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:17.362217 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.335553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:35741)
0808 06:00:17.335831 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.335846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.336285 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.336827 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.336834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.337271 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:17.337759 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.337772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.338820 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.338823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.346736 (+ 7913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.346748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.348285 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.348290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.348423 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.354194 (+ 5771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:17.354206 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:17.357136 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:17.359474 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.359478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.359480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.360304 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.361105 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:17.361107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:17.361108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:17.361175 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:17.361625 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:17.361628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:17.361716 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.361933 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.362063 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1862,"thread_start_us":92,"threads_started":1}
I20260808 06:00:17.362335 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.335689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35741 (local address 127.13.229.126:33945)
0808 06:00:17.335814 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.335818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.335953 (+ 135us) server_negotiation.cc:407] Connection header received
0808 06:00:17.336490 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.336493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.336549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.336649 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.337904 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.338712 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.347603 (+ 8891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.348007 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.348100 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.357329 (+ 9229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.357332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.357334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.357360 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.359208 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.360833 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.360838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.360843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.360947 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.361279 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.361282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.361284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.361511 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.361612 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.362046 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.362163 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:17.363234 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: NON_VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: true } } }
I20260808 06:00:17.363353 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:17.363583 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.348222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.13.229.124:46803)
0808 06:00:17.348787 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.348792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.349154 (+ 362us) server_negotiation.cc:407] Connection header received
0808 06:00:17.349201 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.349207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.349257 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.349340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.350398 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.351420 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.357225 (+ 5805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.357505 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.357618 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.359329 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.359333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.359335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.359359 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.361361 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.361853 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.361857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.361858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.361976 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.362280 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.362283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.362285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.362532 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.362681 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.363268 (+ 587us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.363392 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":81,"threads_started":1}
I20260808 06:00:17.365124 21986 heartbeater.cc:347] Connected to a master server at 127.13.229.126:33945
I20260808 06:00:17.365217 21986 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:17.365384 21986 heartbeater.cc:511] Master 127.13.229.126:33945 requested a full tablet report, sending...
I20260808 06:00:17.366673 20684 ts_manager.cc:194] Re-registered known tserver with Master: 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:17.367353 20684 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:35741
I20260808 06:00:17.371701 22004 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: attempting to promote NON_VOTER f192ba9e88d64933b0ad1b7fb31c4cb8 to VOTER
I20260808 06:00:17.372027 22004 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } }
I20260808 06:00:17.373932 21326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260808 06:00:17.374010 21518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 LEARNER]: Refusing update from remote peer bb01fc67f68e440e95e9f6d0bf4cd8ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260808 06:00:17.374337 22003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 06:00:17.374593 21848 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 06:00:17.376389 22003 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.377941 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.377918 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.378036 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:17.378046 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:17.378180 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.378273 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:17.379374 21326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.379314 21518 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.381393 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.381510 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:17.383085 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:17.383210 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:17.384554 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.378472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41229 (local address 127.0.0.1:43722)
0808 06:00:17.378768 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.378781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.378871 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.380222 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.380226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.380245 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.380343 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.380349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.381841 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.381845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.383050 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.383061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.384160 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.384164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.384251 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.384277 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.384357 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.384422 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260808 06:00:17.384665 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.379141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.13.229.65:41229)
0808 06:00:17.379578 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.379594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.379618 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:00:17.379703 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.379709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.380053 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.380519 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.380655 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.381716 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.383196 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.384035 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.384145 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.384278 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.384390 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.384483 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260808 06:00:17.385502 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.378254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:46725 (local address 127.0.0.1:59950)
0808 06:00:17.378739 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.378753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.378830 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.380859 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.380872 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.380892 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.380966 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.380972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.382564 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.382567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.383642 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.383652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.385059 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.385062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.385204 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.385225 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.385307 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.385382 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":19}
I20260808 06:00:17.385704 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.378461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.13.229.66:46725)
0808 06:00:17.380389 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.380411 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.380436 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:00:17.380512 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.380516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.380706 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.381104 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.381250 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.382438 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.383831 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.384915 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.385173 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.385303 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.385352 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.385433 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1826,"thread_start_us":111,"threads_started":1}
I20260808 06:00:17.387579 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.378550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38289 (local address 127.0.0.1:45670)
0808 06:00:17.382773 (+ 4223us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.382786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.382905 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.383466 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.383469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.383483 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.383545 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.383548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.384933 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.384936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.386392 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.386403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.387194 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.387198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.387287 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.387308 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.387389 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.387458 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4162,"mutex_wait_us":22,"thread_start_us":82,"threads_started":1}
I20260808 06:00:17.387624 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.378811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.13.229.67:38289)
0808 06:00:17.379135 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.379146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.382881 (+ 3735us) server_negotiation.cc:407] Connection header received
0808 06:00:17.383011 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.383015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.383202 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.383361 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.383662 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.384699 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.386516 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.387067 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.387192 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.387316 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.387415 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.387497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":128,"threads_started":1}
I20260808 06:00:17.388013 21627 tablet_service.cc:1467] Processing CreateTablet for tablet f97464801ac34fe898a97637c418de44 (DEFAULT_TABLE table=test-table [id=49abdd95f36c460ba24eaba1db843ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:17.388446 21769 tablet_service.cc:1467] Processing CreateTablet for tablet f97464801ac34fe898a97637c418de44 (DEFAULT_TABLE table=test-table [id=49abdd95f36c460ba24eaba1db843ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:17.389158 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97464801ac34fe898a97637c418de44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:17.389434 21769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97464801ac34fe898a97637c418de44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:17.390137 21909 tablet_service.cc:1467] Processing CreateTablet for tablet f97464801ac34fe898a97637c418de44 (DEFAULT_TABLE table=test-table [id=49abdd95f36c460ba24eaba1db843ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:17.390966 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97464801ac34fe898a97637c418de44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:17.397205 22020 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Bootstrap starting.
I20260808 06:00:17.398110 22022 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Bootstrap starting.
I20260808 06:00:17.398386 22021 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Bootstrap starting.
I20260808 06:00:17.399699 22020 tablet_bootstrap.cc:654] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:17.400436 22021 tablet_bootstrap.cc:654] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:17.400436 22022 tablet_bootstrap.cc:654] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:17.400521 22020 log.cc:824] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:17.401373 22021 log.cc:824] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:17.401365 22022 log.cc:824] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:17.402597 22020 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: No bootstrap required, opened a new log
I20260808 06:00:17.402801 22020 ts_tablet_manager.cc:1397] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:17.403236 22022 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: No bootstrap required, opened a new log
I20260808 06:00:17.403424 22022 ts_tablet_manager.cc:1397] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:00:17.403661 22021 tablet_bootstrap.cc:492] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: No bootstrap required, opened a new log
I20260808 06:00:17.403829 22021 ts_tablet_manager.cc:1397] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:17.406664 22020 raft_consensus.cc:348] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.406941 22020 raft_consensus.cc:374] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:17.407006 22020 raft_consensus.cc:729] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611d3bccd07b40a6a700d4599f375ee7, State: Initialized, Role: FOLLOWER
I20260808 06:00:17.407450 22020 consensus_queue.cc:260] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.408668 22020 ts_tablet_manager.cc:1428] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:00:17.408522 22022 raft_consensus.cc:348] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.408522 22021 raft_consensus.cc:348] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.408751 22021 raft_consensus.cc:374] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:17.408751 22022 raft_consensus.cc:374] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:17.408812 22021 raft_consensus.cc:729] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73778dd88e2a417e86297b0a56f16ddc, State: Initialized, Role: FOLLOWER
I20260808 06:00:17.408812 22022 raft_consensus.cc:729] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab97520046a4864ae49fa62bebaf6c2, State: Initialized, Role: FOLLOWER
I20260808 06:00:17.409041 21986 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
I20260808 06:00:17.409252 22021 consensus_queue.cc:260] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.410117 22021 ts_tablet_manager.cc:1428] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:00:17.410449 21695 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
I20260808 06:00:17.410427 22022 consensus_queue.cc:260] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.411556 22022 ts_tablet_manager.cc:1428] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260808 06:00:17.413061 21838 heartbeater.cc:503] Master 127.13.229.126:33945 was elected leader, sending a full tablet report...
W20260808 06:00:17.584195 21988 tablet.cc:2369] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:00:17.624835 21840 tablet.cc:2369] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:00:17.651439 21697 tablet.cc:2369] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:00:17.662410 22027 raft_consensus.cc:482] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:17.662631 22027 raft_consensus.cc:504] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.663828 22027 leader_election.cc:288] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725), 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:17.668366 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.664193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:46725 (local address 127.13.229.65:50593)
0808 06:00:17.664516 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.664533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.664653 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.664948 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.664951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.664971 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.665152 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.665161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.666091 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.666095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.667205 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.667220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.667857 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.667865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.668006 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.668036 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.668121 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.668182 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":70,"threads_started":1}
I20260808 06:00:17.668412 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.664186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:50593 (local address 127.13.229.66:46725)
0808 06:00:17.664342 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.664347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.664604 (+ 257us) server_negotiation.cc:407] Connection header received
0808 06:00:17.664759 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.664761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.664803 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.664906 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.665346 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.665966 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.667336 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.667736 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.667860 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.667978 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.668158 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.668255 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 06:00:17.668536 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.664442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:45959 (local address 127.13.229.67:38289)
0808 06:00:17.664617 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.664622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.664639 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:00:17.664702 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.664708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.664808 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.664946 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:17.665296 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.666107 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.667186 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.667811 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.667958 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.668015 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.668289 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.668371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 06:00:17.668563 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.664095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38289 (local address 127.13.229.65:45959)
0808 06:00:17.664346 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:17.664363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:17.664477 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:17.665049 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:17.665052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:17.665073 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:17.665166 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.665171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.666208 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.666211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.667021 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:17.667058 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.668043 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.668045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:17.668148 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.668173 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:00:17.668250 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:00:17.668316 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260808 06:00:17.669456 21789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97464801ac34fe898a97637c418de44" candidate_uuid: "73778dd88e2a417e86297b0a56f16ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab97520046a4864ae49fa62bebaf6c2" is_pre_election: true
I20260808 06:00:17.669664 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97464801ac34fe898a97637c418de44" candidate_uuid: "73778dd88e2a417e86297b0a56f16ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "611d3bccd07b40a6a700d4599f375ee7" is_pre_election: true
I20260808 06:00:17.669800 21789 raft_consensus.cc:2454] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73778dd88e2a417e86297b0a56f16ddc in term 0.
I20260808 06:00:17.669998 21934 raft_consensus.cc:2454] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73778dd88e2a417e86297b0a56f16ddc in term 0.
I20260808 06:00:17.670279 21581 leader_election.cc:302] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73778dd88e2a417e86297b0a56f16ddc, aab97520046a4864ae49fa62bebaf6c2; no voters:
I20260808 06:00:17.670538 22027 raft_consensus.cc:2781] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:17.670622 22027 raft_consensus.cc:482] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:17.670665 22027 raft_consensus.cc:3037] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:17.671671 22027 raft_consensus.cc:504] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.672071 22027 leader_election.cc:288] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [CANDIDATE]: Term 1 election: Requested vote from peers aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725), 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289)
I20260808 06:00:17.672423 21789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97464801ac34fe898a97637c418de44" candidate_uuid: "73778dd88e2a417e86297b0a56f16ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab97520046a4864ae49fa62bebaf6c2"
I20260808 06:00:17.672562 21789 raft_consensus.cc:3037] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:17.672618 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97464801ac34fe898a97637c418de44" candidate_uuid: "73778dd88e2a417e86297b0a56f16ddc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "611d3bccd07b40a6a700d4599f375ee7"
I20260808 06:00:17.672765 21934 raft_consensus.cc:3037] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:17.673606 21789 raft_consensus.cc:2454] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73778dd88e2a417e86297b0a56f16ddc in term 1.
I20260808 06:00:17.673887 21581 leader_election.cc:302] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73778dd88e2a417e86297b0a56f16ddc, aab97520046a4864ae49fa62bebaf6c2; no voters:
I20260808 06:00:17.673964 21934 raft_consensus.cc:2454] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73778dd88e2a417e86297b0a56f16ddc in term 1.
I20260808 06:00:17.674096 22027 raft_consensus.cc:2781] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:17.674310 22027 raft_consensus.cc:686] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 1 LEADER]: Becoming Leader. State: Replica: 73778dd88e2a417e86297b0a56f16ddc, State: Running, Role: LEADER
I20260808 06:00:17.674655 22027 consensus_queue.cc:237] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } }
I20260808 06:00:17.678200 20682 catalog_manager.cc:5495] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc reported cstate change: term changed from 0 to 1, leader changed from <none> to 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65). New cstate: current_term: 1 leader_uuid: "73778dd88e2a417e86297b0a56f16ddc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } health_report { overall_health: UNKNOWN } } }
I20260808 06:00:17.745131 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.741607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.13.229.65:41229)
0808 06:00:17.741767 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.741772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.741788 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:17.741884 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.741887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.741943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.742059 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:17.742308 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.742807 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.743672 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.743918 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.744032 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.744339 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:17.744748 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:17.744891 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.744928 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.744983 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:00:17.751362 21934 raft_consensus.cc:1261] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Refusing update from remote peer 73778dd88e2a417e86297b0a56f16ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:17.751358 21789 raft_consensus.cc:1261] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Refusing update from remote peer 73778dd88e2a417e86297b0a56f16ddc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:17.752136 22031 consensus_queue.cc:1035] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:17.752375 22027 consensus_queue.cc:1035] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [LEADER]: Connected to new peer: Peer: permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:17.760300 22034 mvcc.cc:205] Tried to move back new op lower bound from 7316147477502009344 to 7316147477197787136. Current Snapshot: MvccSnapshot[applied={T|T < 7316147477502009344}]
I20260808 06:00:17.761638 22035 mvcc.cc:205] Tried to move back new op lower bound from 7316147477502009344 to 7316147477197787136. Current Snapshot: MvccSnapshot[applied={T|T < 7316147477502009344}]
I20260808 06:00:17.765954 22036 mvcc.cc:205] Tried to move back new op lower bound from 7316147477502009344 to 7316147477197787136. Current Snapshot: MvccSnapshot[applied={T|T < 7316147477502009344 or (T in {7316147477502009344})}]
I20260808 06:00:17.769690 20685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:17.769819 20685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:17.774096 20685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb01fc67f68e440e95e9f6d0bf4cd8ff: Sending DeleteTablet for 3 replicas of tablet f97464801ac34fe898a97637c418de44
I20260808 06:00:17.774959 21769 tablet_service.cc:1514] Processing DeleteTablet for tablet f97464801ac34fe898a97637c418de44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:17 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:59950
I20260808 06:00:17.775614 22044 tablet_replica.cc:330] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: stopping tablet replica
I20260808 06:00:17.775640 21627 tablet_service.cc:1514] Processing DeleteTablet for tablet f97464801ac34fe898a97637c418de44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:17 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:43722
I20260808 06:00:17.775981 22044 raft_consensus.cc:2229] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:17.776319 22044 raft_consensus.cc:2258] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:17.777149 21909 tablet_service.cc:1514] Processing DeleteTablet for tablet f97464801ac34fe898a97637c418de44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:17 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:45670
I20260808 06:00:17.777215 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 20649
I20260808 06:00:17.778946 22044 ts_tablet_manager.cc:1905] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:17.779445 22047 tablet_replica.cc:330] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: stopping tablet replica
I20260808 06:00:17.779719 22046 tablet_replica.cc:330] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: stopping tablet replica
I20260808 06:00:17.780129 22047 raft_consensus.cc:2229] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:17.780453 22046 raft_consensus.cc:2229] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:17.780864 22046 raft_consensus.cc:2258] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:17.781282 22047 raft_consensus.cc:2258] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:17.782680 22044 ts_tablet_manager.cc:1918] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:17.782791 22044 log.cc:1196] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/f97464801ac34fe898a97637c418de44
I20260808 06:00:17.783165 22044 ts_tablet_manager.cc:1939] T f97464801ac34fe898a97637c418de44 P aab97520046a4864ae49fa62bebaf6c2: Deleting consensus metadata
I20260808 06:00:17.784225 22047 ts_tablet_manager.cc:1905] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:17.784586 22046 ts_tablet_manager.cc:1905] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:17.787606 22046 ts_tablet_manager.cc:1918] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 06:00:17.788985 21721 connection.cc:535] server connection from 127.0.0.1:59950 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 06:00:17.789546 21303 connection.cc:535] server connection from 127.0.0.1:57972 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:00:17.791532 22047 ts_tablet_manager.cc:1918] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:17.791704 22046 log.cc:1196] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/f97464801ac34fe898a97637c418de44
I20260808 06:00:17.792210 22047 log.cc:1196] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/f97464801ac34fe898a97637c418de44
I20260808 06:00:17.792762 22046 ts_tablet_manager.cc:1939] T f97464801ac34fe898a97637c418de44 P 73778dd88e2a417e86297b0a56f16ddc: Deleting consensus metadata
I20260808 06:00:17.793454 22047 ts_tablet_manager.cc:1939] T f97464801ac34fe898a97637c418de44 P 611d3bccd07b40a6a700d4599f375ee7: Deleting consensus metadata
W20260808 06:00:17.794598 21581 connection.cc:441] server connection from 127.0.0.1:43722 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43722 (request call id 1) could send its response
W20260808 06:00:17.795774 21866 connection.cc:441] server connection from 127.0.0.1:45670 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45670 (request call id 1) could send its response
I20260808 06:00:17.802182 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.792358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.13.229.125:35829)
0808 06:00:17.792491 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.792495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.792966 (+ 471us) server_negotiation.cc:407] Connection header received
0808 06:00:17.793072 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.793075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.793119 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.793196 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.794316 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.795108 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.796309 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.796670 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.796792 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.798343 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.798347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.798350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.798377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.800142 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.800802 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.800806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.800808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.800896 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.801205 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.801208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.801210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.801421 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.801525 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.801887 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.802021 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 06:00:17.803463 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.776092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.13.229.125:35829)
0808 06:00:17.776208 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.776213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.784682 (+ 8469us) server_negotiation.cc:407] Connection header received
0808 06:00:17.784815 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.784819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.784872 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.784959 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.785992 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.786891 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.788302 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.788733 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.788826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.800223 (+ 11397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.800227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.800229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.800254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.801823 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.802287 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.802290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.802292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.802365 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.802622 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.802625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.802627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.802799 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.802904 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.803224 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.803325 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:00:17.813917 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.804445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.13.229.124:46803)
0808 06:00:17.804548 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.804552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.804565 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:17.804602 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.804604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.804646 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.804723 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.805501 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.806354 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.807943 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.808668 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.808777 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.810382 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.810386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.810388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.810410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.812242 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.812709 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.812712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.812714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.812792 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.813051 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.813055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.813057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.813223 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.813378 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.813611 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.813715 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":327,"server-negotiator.queue_time_us":41}
I20260808 06:00:17.814154 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:17.776104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.13.229.124:46803)
0808 06:00:17.776205 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:17.776209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:17.805145 (+ 28936us) server_negotiation.cc:407] Connection header received
0808 06:00:17.805282 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:17.805285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:17.805322 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:17.805364 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:17.806220 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.806982 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.807974 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:17.808286 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:17.808406 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:17.810119 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:17.810123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:17.810126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:17.810149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:17.811935 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.812403 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.812408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.812410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.812483 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:17.812793 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:17.812798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:17.812800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:17.813031 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:17.813129 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:17.813663 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 06:00:17.813823 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":37}
I20260808 06:00:18.767242 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:18.766779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:39443)
0808 06:00:18.766997 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:18.767117 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260808 06:00:18.767412 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:18.766644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:57393)
0808 06:00:18.767195 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:18.767274 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":89,"threads_started":1}
I20260808 06:00:18.767480 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:18.766940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:56093)
0808 06:00:18.767162 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:18.767238 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
W20260808 06:00:18.767541 21695 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:33945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
W20260808 06:00:18.767705 21838 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:33945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
W20260808 06:00:18.767975 21986 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:33945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.274801 22058 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb01fc67f68e440e95e9f6d0bf4cd8ff)
I20260808 06:00:19.274997 22058 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } }
I20260808 06:00:19.275350 22059 raft_consensus.cc:482] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb01fc67f68e440e95e9f6d0bf4cd8ff)
I20260808 06:00:19.275534 22059 raft_consensus.cc:504] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } }
I20260808 06:00:19.276131 22058 leader_election.cc:288] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945), f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124:46803)
I20260808 06:00:19.276542 22059 leader_election.cc:288] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945), d33930902ee24abfb32c72e2cc38a77e (127.13.229.125:35829)
I20260808 06:00:19.276847 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.276277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45224)
0808 06:00:19.276619 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.276720 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
W20260808 06:00:19.277035 21300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.277217 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.276718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45230)
0808 06:00:19.276884 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.277059 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":46,"threads_started":1}
W20260808 06:00:19.277478 21491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.278162 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.277898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45246)
0808 06:00:19.278037 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.278086 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260808 06:00:19.278291 21300 leader_election.cc:334] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945): Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.278596 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.278238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45250)
0808 06:00:19.278505 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.278548 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260808 06:00:19.278712 21491 leader_election.cc:334] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945): Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.280788 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.276499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.13.229.124:46803)
0808 06:00:19.276760 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.276767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.276982 (+ 215us) server_negotiation.cc:407] Connection header received
0808 06:00:19.277035 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.277039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.277093 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.277172 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.277596 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.278382 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.279712 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.280325 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.280453 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.280576 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.280647 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.280706 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":186,"thread_start_us":55,"threads_started":1}
I20260808 06:00:19.280931 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.276395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:46803 (local address 127.0.0.1:39100)
0808 06:00:19.276758 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.276770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.276883 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.277321 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.277324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.277345 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.277461 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.277468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.278487 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.278490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.279595 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.279604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.280445 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.280448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.280518 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.280540 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.280639 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.280806 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":81,"thread_start_us":71,"threads_started":1}
I20260808 06:00:19.281366 21518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33930902ee24abfb32c72e2cc38a77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" is_pre_election: true
I20260808 06:00:19.281562 21518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33930902ee24abfb32c72e2cc38a77e in term 1.
I20260808 06:00:19.281796 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.277259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.13.229.125:35829)
0808 06:00:19.277553 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.277563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.277728 (+ 165us) server_negotiation.cc:407] Connection header received
0808 06:00:19.277810 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.277813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.277853 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.277927 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.278369 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.279194 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.280618 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.281254 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.281378 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.281421 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.281616 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.281680 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
I20260808 06:00:19.281811 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.276985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:35829 (local address 127.0.0.1:58020)
0808 06:00:19.277539 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.277554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.277654 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.278071 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.278075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.278092 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.278216 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.278224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.279322 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.279325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.280499 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.280511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.281373 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.281377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.281480 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.281524 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.281598 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.281671 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260808 06:00:19.281949 21301 leader_election.cc:302] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d33930902ee24abfb32c72e2cc38a77e, f192ba9e88d64933b0ad1b7fb31c4cb8; no voters: bb01fc67f68e440e95e9f6d0bf4cd8ff
I20260808 06:00:19.282198 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 06:00:19.282173 21326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d33930902ee24abfb32c72e2cc38a77e" is_pre_election: true
I20260808 06:00:19.282261 22058 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bb01fc67f68e440e95e9f6d0bf4cd8ff)
I20260808 06:00:19.282333 22058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:00:19.283380 22058 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } }
I20260808 06:00:19.283624 21326 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f192ba9e88d64933b0ad1b7fb31c4cb8 in current term 2: Already voted for candidate d33930902ee24abfb32c72e2cc38a77e in this term.
I20260808 06:00:19.283766 22058 leader_election.cc:288] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 election: Requested vote from peers bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945), f192ba9e88d64933b0ad1b7fb31c4cb8 (127.13.229.124:46803)
I20260808 06:00:19.284192 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.283841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45254)
0808 06:00:19.284079 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.284120 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260808 06:00:19.284255 21518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d33930902ee24abfb32c72e2cc38a77e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8"
I20260808 06:00:19.284364 21518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:00:19.284368 21491 leader_election.cc:302] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f192ba9e88d64933b0ad1b7fb31c4cb8; no voters: bb01fc67f68e440e95e9f6d0bf4cd8ff, d33930902ee24abfb32c72e2cc38a77e
I20260808 06:00:19.285375 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.285133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45266)
0808 06:00:19.285253 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.285309 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:00:19.285518 21300 leader_election.cc:334] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945): Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
I20260808 06:00:19.285658 21518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33930902ee24abfb32c72e2cc38a77e in term 2.
I20260808 06:00:19.285789 22059 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:00:19.285946 21301 leader_election.cc:302] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d33930902ee24abfb32c72e2cc38a77e, f192ba9e88d64933b0ad1b7fb31c4cb8; no voters: bb01fc67f68e440e95e9f6d0bf4cd8ff
I20260808 06:00:19.286114 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:00:19.286326 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [term 2 LEADER]: Becoming Leader. State: Replica: d33930902ee24abfb32c72e2cc38a77e, State: Running, Role: LEADER
I20260808 06:00:19.286576 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } }
I20260808 06:00:19.287544 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33930902ee24abfb32c72e2cc38a77e. Latest consensus state: current_term: 2 leader_uuid: "d33930902ee24abfb32c72e2cc38a77e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:19.287638 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:19.288002 22070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:19.290198 22070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:19.290597 22070 catalog_manager.cc:1259] Loaded cluster ID: ae1d44125a27488caafaddc63bca4f1b
I20260808 06:00:19.290668 22070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:19.291214 22070 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:19.291608 22070 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Loaded TSK: 0
I20260808 06:00:19.292335 22070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:19.325615 21316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:19.328792 21316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:19.331516 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.331244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45288)
0808 06:00:19.331380 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.331440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 06:00:19.331625 21518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [term 2 FOLLOWER]: Refusing update from remote peer d33930902ee24abfb32c72e2cc38a77e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 06:00:19.332222 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.331987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.0.0.1:45294)
0808 06:00:19.332116 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.332166 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 06:00:19.332312 21300 consensus_peers.cc:487] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e -> Peer bb01fc67f68e440e95e9f6d0bf4cd8ff (127.13.229.126:33945): Couldn't send request to peer bb01fc67f68e440e95e9f6d0bf4cd8ff. Status: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 06:00:19.332436 22068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 06:00:19.333848 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33930902ee24abfb32c72e2cc38a77e. Latest consensus state: current_term: 2 leader_uuid: "d33930902ee24abfb32c72e2cc38a77e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:19.333993 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:19.335198 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d33930902ee24abfb32c72e2cc38a77e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:19.335513 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:19.335583 22072 mvcc.cc:205] Tried to move back new op lower bound from 7316147483977572352 to 7316147483799912448. Current Snapshot: MvccSnapshot[applied={T|T < 7316147483977572352}]
I20260808 06:00:19.335847 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d33930902ee24abfb32c72e2cc38a77e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bb01fc67f68e440e95e9f6d0bf4cd8ff" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33945 } } peers { permanent_uuid: "d33930902ee24abfb32c72e2cc38a77e" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 35829 } attrs { promote: false } } peers { permanent_uuid: "f192ba9e88d64933b0ad1b7fb31c4cb8" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 46803 } attrs { promote: false } } }
I20260808 06:00:19.335997 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f192ba9e88d64933b0ad1b7fb31c4cb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:00:19.345804 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.341583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41229 (local address 127.0.0.1:43738)
0808 06:00:19.341711 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.341726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.341806 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.342340 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.342343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.342358 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.342466 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.342472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343451 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.343454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.344540 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.344551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345412 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.345505 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345526 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.345613 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.345680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 06:00:19.345816 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.341741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.13.229.65:41229)
0808 06:00:19.342066 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.342069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.342081 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:00:19.342124 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.342126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.342195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.342332 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.342575 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343342 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.344667 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345283 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345400 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345458 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.345631 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.345700 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
I20260808 06:00:19.346124 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.340995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:46725 (local address 127.0.0.1:59952)
0808 06:00:19.341135 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.341150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.341234 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.342196 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.342200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.342220 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.342339 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.342345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343491 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.343495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.344312 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.344321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345782 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.345881 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345905 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.345967 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.346026 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":1}
I20260808 06:00:19.346123 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.341704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.13.229.67:38289)
0808 06:00:19.341943 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.341949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.342055 (+ 106us) server_negotiation.cc:407] Connection header received
0808 06:00:19.342233 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.342236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.342282 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.342333 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.342644 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343184 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345148 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345557 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345676 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345732 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.345910 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.345998 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260808 06:00:19.346427 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.341620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38289 (local address 127.0.0.1:45680)
0808 06:00:19.341982 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.341994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.342114 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.342457 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.342460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.342473 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.342559 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.342564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343300 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.343305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.345032 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.345040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345675 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.345765 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345786 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.346247 (+ 461us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.346320 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":739,"thread_start_us":85,"threads_started":1}
I20260808 06:00:19.346490 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.341100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.13.229.66:46725)
0808 06:00:19.341948 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.341952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.341965 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:19.342022 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.342028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.342085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.342172 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.342468 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.343342 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.344437 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.345031 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.345137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.345199 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.346276 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.346356 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":104,"threads_started":1}
I20260808 06:00:19.346540 21627 tablet_service.cc:1467] Processing CreateTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 (DEFAULT_TABLE table=test-table [id=7a8f250069844b35b4d323e7e4da4cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:19.346855 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c63cacfe6d40c69f798ef3613b36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:19.346976 21909 tablet_service.cc:1467] Processing CreateTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 (DEFAULT_TABLE table=test-table [id=7a8f250069844b35b4d323e7e4da4cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:19.347312 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c63cacfe6d40c69f798ef3613b36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:19.348536 21769 tablet_service.cc:1467] Processing CreateTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 (DEFAULT_TABLE table=test-table [id=7a8f250069844b35b4d323e7e4da4cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:19.348839 21769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c63cacfe6d40c69f798ef3613b36b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:19.349468 22081 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: Bootstrap starting.
I20260808 06:00:19.350394 22081 tablet_bootstrap.cc:654] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:19.351291 22082 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Bootstrap starting.
I20260808 06:00:19.351622 22081 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: No bootstrap required, opened a new log
I20260808 06:00:19.351630 22084 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Bootstrap starting.
I20260808 06:00:19.351727 22081 ts_tablet_manager.cc:1397] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:19.352247 22082 tablet_bootstrap.cc:654] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:19.352193 22081 raft_consensus.cc:348] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.352308 22081 raft_consensus.cc:374] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:19.352348 22081 raft_consensus.cc:729] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73778dd88e2a417e86297b0a56f16ddc, State: Initialized, Role: FOLLOWER
I20260808 06:00:19.352504 22084 tablet_bootstrap.cc:654] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:19.352496 22081 consensus_queue.cc:260] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.352873 22081 ts_tablet_manager.cc:1428] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:19.353665 22082 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: No bootstrap required, opened a new log
I20260808 06:00:19.353770 22082 ts_tablet_manager.cc:1397] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:19.354139 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.353295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:33431)
0808 06:00:19.353998 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.354054 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":101,"threads_started":1}
I20260808 06:00:19.354156 22082 raft_consensus.cc:348] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.354290 22082 raft_consensus.cc:374] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:19.354329 22082 raft_consensus.cc:729] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611d3bccd07b40a6a700d4599f375ee7, State: Initialized, Role: FOLLOWER
I20260808 06:00:19.354447 22082 consensus_queue.cc:260] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.354780 22082 ts_tablet_manager.cc:1428] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:19.355413 22084 tablet_bootstrap.cc:492] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: No bootstrap required, opened a new log
I20260808 06:00:19.355466 21696 heartbeater.cc:503] Master 127.13.229.125:35829 was elected leader, sending a full tablet report...
I20260808 06:00:19.355499 22084 ts_tablet_manager.cc:1397] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 06:00:19.356057 22084 raft_consensus.cc:348] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.356266 22084 raft_consensus.cc:374] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:19.356318 22084 raft_consensus.cc:729] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab97520046a4864ae49fa62bebaf6c2, State: Initialized, Role: FOLLOWER
I20260808 06:00:19.356524 22084 consensus_queue.cc:260] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.356894 22084 ts_tablet_manager.cc:1428] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 06:00:19.357242 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.354503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:45311)
0808 06:00:19.354765 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.354814 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260808 06:00:19.358160 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.357636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:33273)
0808 06:00:19.358015 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.358068 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":90,"threads_started":1}
I20260808 06:00:19.358532 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.355636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:33627)
0808 06:00:19.358376 (+ 2740us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.358435 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2676,"thread_start_us":104,"threads_started":1}
I20260808 06:00:19.358824 21839 heartbeater.cc:503] Master 127.13.229.125:35829 was elected leader, sending a full tablet report...
I20260808 06:00:19.358934 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.358540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:50797)
0808 06:00:19.358760 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.358813 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260808 06:00:19.359746 21987 heartbeater.cc:503] Master 127.13.229.125:35829 was elected leader, sending a full tablet report...
I20260808 06:00:19.359846 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.359587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:58263)
0808 06:00:19.359677 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.359726 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 06:00:19.476048 22090 raft_consensus.cc:482] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:19.476197 22090 raft_consensus.cc:504] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.477315 22090 leader_election.cc:288] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725), 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
I20260808 06:00:19.481598 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.477544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:46725 (local address 127.13.229.67:57023)
0808 06:00:19.477674 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.477690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.477799 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.478247 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.478250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.478264 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.478421 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.478428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.479387 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.479390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.480408 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.480419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.481176 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.481182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.481279 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.481310 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.481392 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.481445 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 06:00:19.481734 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.477954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:59303 (local address 127.13.229.65:41229)
0808 06:00:19.478080 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.478084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.478344 (+ 260us) server_negotiation.cc:407] Connection header received
0808 06:00:19.478409 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.478412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.478467 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.478568 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.478881 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.479483 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.480689 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.481085 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.481226 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.481281 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.481527 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.481604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:00:19.481716 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.477672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:57023 (local address 127.13.229.66:46725)
0808 06:00:19.477821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.477829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.477851 (+ 22us) server_negotiation.cc:407] Connection header received
0808 06:00:19.477942 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.477946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.478131 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.478236 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:19.478572 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.479268 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.480557 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.481038 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.481149 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.481234 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.481436 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.481529 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:00:19.482177 21789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c63cacfe6d40c69f798ef3613b36b2" candidate_uuid: "611d3bccd07b40a6a700d4599f375ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab97520046a4864ae49fa62bebaf6c2" is_pre_election: true
I20260808 06:00:19.482251 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.477818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41229 (local address 127.13.229.67:59303)
0808 06:00:19.478127 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.478139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.478251 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.478607 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:19.478611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:19.478633 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:19.478738 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.478745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.479600 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.479603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.480546 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:19.480558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.481253 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.481260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:19.481356 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.481411 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 06:00:19.481499 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 06:00:19.481570 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":149,"threads_started":1}
I20260808 06:00:19.482369 21789 raft_consensus.cc:2454] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611d3bccd07b40a6a700d4599f375ee7 in term 0.
I20260808 06:00:19.482776 21866 leader_election.cc:302] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 611d3bccd07b40a6a700d4599f375ee7, aab97520046a4864ae49fa62bebaf6c2; no voters:
I20260808 06:00:19.482869 21647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c63cacfe6d40c69f798ef3613b36b2" candidate_uuid: "611d3bccd07b40a6a700d4599f375ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73778dd88e2a417e86297b0a56f16ddc" is_pre_election: true
I20260808 06:00:19.483080 22090 raft_consensus.cc:2781] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:19.483144 22090 raft_consensus.cc:482] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:19.483111 21647 raft_consensus.cc:2454] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611d3bccd07b40a6a700d4599f375ee7 in term 0.
I20260808 06:00:19.483176 22090 raft_consensus.cc:3037] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:19.484483 22090 raft_consensus.cc:504] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.484866 22090 leader_election.cc:288] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [CANDIDATE]: Term 1 election: Requested vote from peers aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725), 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229)
I20260808 06:00:19.485268 21789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c63cacfe6d40c69f798ef3613b36b2" candidate_uuid: "611d3bccd07b40a6a700d4599f375ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab97520046a4864ae49fa62bebaf6c2"
I20260808 06:00:19.485373 21647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c63cacfe6d40c69f798ef3613b36b2" candidate_uuid: "611d3bccd07b40a6a700d4599f375ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73778dd88e2a417e86297b0a56f16ddc"
I20260808 06:00:19.485426 21789 raft_consensus.cc:3037] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:19.485527 21647 raft_consensus.cc:3037] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:19.486413 21789 raft_consensus.cc:2454] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611d3bccd07b40a6a700d4599f375ee7 in term 1.
I20260808 06:00:19.486574 21647 raft_consensus.cc:2454] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611d3bccd07b40a6a700d4599f375ee7 in term 1.
I20260808 06:00:19.486713 21866 leader_election.cc:302] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 611d3bccd07b40a6a700d4599f375ee7, aab97520046a4864ae49fa62bebaf6c2; no voters:
I20260808 06:00:19.486886 22090 raft_consensus.cc:2781] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:19.487108 22090 raft_consensus.cc:686] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 611d3bccd07b40a6a700d4599f375ee7, State: Running, Role: LEADER
I20260808 06:00:19.487396 22090 consensus_queue.cc:237] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } }
I20260808 06:00:19.487797 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.487555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:36525)
0808 06:00:19.487668 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.487723 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 06:00:19.489952 21317 catalog_manager.cc:5495] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "611d3bccd07b40a6a700d4599f375ee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "611d3bccd07b40a6a700d4599f375ee7" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 } health_report { overall_health: UNKNOWN } } }
I20260808 06:00:19.533742 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.530195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.13.229.67:38289)
0808 06:00:19.530366 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:19.530370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:19.530383 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:19.530461 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:19.530463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:19.530518 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:19.530634 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:19.530901 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.531406 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.532268 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:19.532492 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:19.532606 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:19.532881 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:19.533325 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:19.533452 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:19.533526 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 06:00:19.533582 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 06:00:19.536754 21789 raft_consensus.cc:1261] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Refusing update from remote peer 611d3bccd07b40a6a700d4599f375ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:19.537092 21647 raft_consensus.cc:1261] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Refusing update from remote peer 611d3bccd07b40a6a700d4599f375ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:19.537382 22090 consensus_queue.cc:1035] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab97520046a4864ae49fa62bebaf6c2" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 46725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:19.537690 22097 consensus_queue.cc:1035] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73778dd88e2a417e86297b0a56f16ddc" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:19.539438 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.539108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:40999)
0808 06:00:19.539269 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.539350 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 06:00:19.540181 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.539945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:60173)
0808 06:00:19.540046 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.540107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 06:00:19.542948 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.542584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:51661)
0808 06:00:19.542826 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.542881 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260808 06:00:19.543073 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.542814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.65:49121)
0808 06:00:19.542952 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.543004 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 06:00:19.547755 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.547494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.67:54757)
0808 06:00:19.547632 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.547686 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 06:00:19.549319 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:19.548524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33945 (local address 127.13.229.66:40409)
0808 06:00:19.549197 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.549249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
I20260808 06:00:19.551644 21315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:19.551803 21315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:19.555399 21315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d33930902ee24abfb32c72e2cc38a77e: Sending DeleteTablet for 3 replicas of tablet c4c63cacfe6d40c69f798ef3613b36b2
I20260808 06:00:19.556046 21769 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:19 UTC) from {username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:59952
I20260808 06:00:19.556345 21627 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:19 UTC) from {username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:43738
I20260808 06:00:19.556370 21909 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c63cacfe6d40c69f798ef3613b36b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:19 UTC) from {username='oryx', principal='oryx/127.13.229.125@KRBTEST.COM'} at 127.0.0.1:45680
I20260808 06:00:19.556682 22112 tablet_replica.cc:330] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc: stopping tablet replica
I20260808 06:00:19.556797 22112 raft_consensus.cc:2229] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:19.556905 22112 raft_consensus.cc:2258] T c4c63cacfe6d40c69f798ef3613b36b2 P 73778dd88e2a417e86297b0a56f16ddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:19.557148 22110 tablet_replica.cc:330] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: stopping tablet replica
I20260808 06:00:19.557327 22110 raft_consensus.cc:2229] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:19.557502 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21469
I20260808 06:00:19.557510 22110 raft_consensus.cc:2258] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:19.558463 22110 ts_tablet_manager.cc:1905] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:19.560976 22110 ts_tablet_manager.cc:1918] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:19.561059 22110 log.cc:1196] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/c4c63cacfe6d40c69f798ef3613b36b2
I20260808 06:00:19.561353 22110 ts_tablet_manager.cc:1939] T c4c63cacfe6d40c69f798ef3613b36b2 P aab97520046a4864ae49fa62bebaf6c2: Deleting consensus metadata
I20260808 06:00:19.562076 21303 catalog_manager.cc:4841] TS aab97520046a4864ae49fa62bebaf6c2 (127.13.229.66:46725): tablet c4c63cacfe6d40c69f798ef3613b36b2 (table test-table [id=7a8f250069844b35b4d323e7e4da4cc8]) successfully deleted
I20260808 06:00:19.565253 22111 tablet_replica.cc:330] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: stopping tablet replica
I20260808 06:00:19.565407 22111 raft_consensus.cc:2229] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:19.565721 22111 raft_consensus.cc:2258] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:19.566491 22111 ts_tablet_manager.cc:1905] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 06:00:19.569535 22078 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:19.568761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41229 (local address 127.0.0.1:43754)
0808 06:00:19.568886 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:19.568902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:19.568985 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:19.569393 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:41229: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 06:00:19.569568 22111 ts_tablet_manager.cc:1918] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:19.569636 22111 log.cc:1196] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/c4c63cacfe6d40c69f798ef3613b36b2
W20260808 06:00:19.569738 21303 catalog_manager.cc:4584] TS 73778dd88e2a417e86297b0a56f16ddc (127.13.229.65:41229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4c63cacfe6d40c69f798ef3613b36b2: Network error: Client connection negotiation failed: client connection to 127.13.229.65:41229: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:00:19.569952 22111 ts_tablet_manager.cc:1939] T c4c63cacfe6d40c69f798ef3613b36b2 P 611d3bccd07b40a6a700d4599f375ee7: Deleting consensus metadata
I20260808 06:00:19.570363 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21702
I20260808 06:00:19.572880 21300 catalog_manager.cc:4841] TS 611d3bccd07b40a6a700d4599f375ee7 (127.13.229.67:38289): tablet c4c63cacfe6d40c69f798ef3613b36b2 (table test-table [id=7a8f250069844b35b4d323e7e4da4cc8]) successfully deleted
I20260808 06:00:19.580696 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21844
I20260808 06:00:19.587478 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21173
I20260808 06:00:19.594151 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 21286
2026-08-08T06:00:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5696 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:19 dist-test-slave-zkpd krb5kdc[22115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:19 dist-test-slave-zkpd krb5kdc[22115](info): set up 2 sockets
Aug 08 06:00:19 dist-test-slave-zkpd krb5kdc[22115](info): commencing operation
krb5kdc: starting...
W20260808 06:00:20.675540 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:20 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab.
Entry for principal oryx/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab.
I20260808 06:00:20.802968 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43329
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab
--principal=oryx/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:20.913676 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:20.914006 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:20.914083 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:20.918255 22130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:20.918321 22130 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 06:00:20.918351 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:20.918408 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:20.918459 22130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:20.918497 22130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:20.923717 22130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:43329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.126.keytab
--principal=oryx/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:20.924755 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:20.925734 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:20.934242 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:20.934255 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:20.934263 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:20.934518 22130 server_base.cc:1034] running on GCE node
I20260808 06:00:20.935145 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:20.936259 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:20.937449 22130 hybrid_clock.cc:648] HybridClock initialized: now 1786168820937431 us; error 32 us; skew 500 ppm
Aug 08 06:00:20 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168820, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:20.940164 22130 init.cc:377] Logged in from keytab as oryx/127.13.229.126@KRBTEST.COM (short username oryx)
I20260808 06:00:20.941046 22130 webserver.cc:460] Webserver started at http://127.13.229.126:34671/ using document root <none> and password file <none>
I20260808 06:00:20.941587 22130 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:20.941676 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:20.941911 22130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:20.943987 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "384ebca1d49748cc97568a0a27be882b"
format_stamp: "Formatted at 2026-08-08 06:00:20 on dist-test-slave-zkpd"
server_key: "94697bf934b9a0c18acb4e6d3201420e"
server_key_iv: "bdab52feb5f516a53b0dd557018a3b7a"
server_key_version: "encryptionkey@0"
I20260808 06:00:20.944556 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "384ebca1d49748cc97568a0a27be882b"
format_stamp: "Formatted at 2026-08-08 06:00:20 on dist-test-slave-zkpd"
server_key: "94697bf934b9a0c18acb4e6d3201420e"
server_key_iv: "bdab52feb5f516a53b0dd557018a3b7a"
server_key_version: "encryptionkey@0"
I20260808 06:00:20.948233 22130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:00:20.950631 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:20.951843 22130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:20.952034 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "384ebca1d49748cc97568a0a27be882b"
format_stamp: "Formatted at 2026-08-08 06:00:20 on dist-test-slave-zkpd"
server_key: "94697bf934b9a0c18acb4e6d3201420e"
server_key_iv: "bdab52feb5f516a53b0dd557018a3b7a"
server_key_version: "encryptionkey@0"
I20260808 06:00:20.952153 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:20.964949 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:20.967784 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:20.968012 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:20.976181 22130 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:43329
I20260808 06:00:20.976207 22207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:43329 every 8 connection(s)
I20260808 06:00:20.977095 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:20.979008 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22130
I20260808 06:00:20.979158 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:20.979450 14229 external_mini_cluster.cc:1428] Setting key be4351d31e938aeba0e16447182b6824
I20260808 06:00:20.979856 22208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:20.985924 22208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Bootstrap starting.
Aug 08 06:00:20 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:20.988191 22208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:20.988894 22208 log.cc:824] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:20.991180 22208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: No bootstrap required, opened a new log
I20260808 06:00:20.994424 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:20.980921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.13.229.126:43329)
0808 06:00:20.981367 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:20.981382 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:20.981417 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:00:20.982069 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:20.982086 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:20.982416 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:20.982757 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:20.983555 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:20.984600 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:20.985442 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:20.985912 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:20.986084 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:20.988625 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:20.988630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:20.988644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:20.988671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:20.991244 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:20.991793 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:20.991798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:20.991804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:20.991972 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:20.992255 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:20.992259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:20.992260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:20.992757 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:20.992946 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:20.993164 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 06:00:20.993478 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":131,"threads_started":1}
I20260808 06:00:20.995177 22208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } }
I20260808 06:00:20.995436 22208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:20.995500 22208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384ebca1d49748cc97568a0a27be882b, State: Initialized, Role: FOLLOWER
I20260808 06:00:20.996030 22208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } }
I20260808 06:00:20.996194 22208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:20.996260 22208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:20.996361 22208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:20.997355 22208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } }
I20260808 06:00:20.997704 22208 leader_election.cc:302] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384ebca1d49748cc97568a0a27be882b; no voters:
I20260808 06:00:20.998004 22208 leader_election.cc:288] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:20.998157 22214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:20.998396 22214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [term 1 LEADER]: Becoming Leader. State: Replica: 384ebca1d49748cc97568a0a27be882b, State: Running, Role: LEADER
I20260808 06:00:20.998690 22214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } }
I20260808 06:00:20.999253 22208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:21.000571 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384ebca1d49748cc97568a0a27be882b. Latest consensus state: current_term: 1 leader_uuid: "384ebca1d49748cc97568a0a27be882b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } } }
I20260808 06:00:21.000567 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384ebca1d49748cc97568a0a27be882b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ebca1d49748cc97568a0a27be882b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 43329 } } }
I20260808 06:00:21.000746 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:21.000864 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:21.002070 22226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:21.004666 22226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:21.010340 22226 catalog_manager.cc:1347] Generated new cluster ID: 0f71f5842e444643b57431e5e179d1e3
I20260808 06:00:21.010412 22226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:21.027032 22226 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:21.027948 22226 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:21.042068 22226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Generated new TSK 0
I20260808 06:00:21.042788 22226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
I20260808 06:00:21.085065 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43329
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:21.204111 22236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:21.204347 22236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:21.204404 22236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:21.208189 22236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:21.208232 22236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:21.208268 22236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:00:21.208299 22236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:21.213249 22236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:21.214169 22236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:21.215106 22236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:21.224555 22242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.224586 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.224606 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:21.224795 22236 server_base.cc:1034] running on GCE node
I20260808 06:00:21.225238 22236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:21.225839 22236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:21.227260 22236 hybrid_clock.cc:648] HybridClock initialized: now 1786168821227224 us; error 47 us; skew 500 ppm
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:21.230183 22236 init.cc:377] Logged in from keytab as oryx/127.13.229.65@KRBTEST.COM (short username oryx)
I20260808 06:00:21.231067 22236 webserver.cc:460] Webserver started at http://127.13.229.65:36401/ using document root <none> and password file <none>
I20260808 06:00:21.231652 22236 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:21.231817 22236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:21.232079 22236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:21.234126 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "8abff3f82b3946feabc50370a0c01450"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "beacc3ab9691233be5710ab9e6137a4a"
server_key_iv: "0d9b462b44470ba2427994c2db5ad778"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.234697 22236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "8abff3f82b3946feabc50370a0c01450"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "beacc3ab9691233be5710ab9e6137a4a"
server_key_iv: "0d9b462b44470ba2427994c2db5ad778"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.238364 22236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 06:00:21.240881 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.242201 22236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:21.242337 22236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "8abff3f82b3946feabc50370a0c01450"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "beacc3ab9691233be5710ab9e6137a4a"
server_key_iv: "0d9b462b44470ba2427994c2db5ad778"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.242447 22236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:21.264986 22236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:21.267824 22236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:21.268045 22236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:21.269428 22236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:21.269506 22236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.269580 22236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:21.269635 22236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:21.280591 22236 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:38477
I20260808 06:00:21.280679 22375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:38477 every 8 connection(s)
I20260808 06:00:21.281677 22236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:21.282542 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22236
I20260808 06:00:21.282693 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:21.283067 14229 external_mini_cluster.cc:1428] Setting key 9486e981bcbb0911cf5b2093cc395060
I20260808 06:00:21.285168 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.270372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40917 (local address 127.13.229.126:43329)
0808 06:00:21.270499 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.270503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.271593 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 06:00:21.272742 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.272747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.272818 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.272883 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.274229 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.275150 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.276562 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.276986 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.277120 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.280296 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.280300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.280305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.280337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.282265 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.282947 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.282952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.282954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.283051 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.283820 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.283826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.283830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.284104 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.284192 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.284819 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.284957 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:00:21.285671 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.270887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.65:40917)
0808 06:00:21.271394 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.271436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.272479 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.273204 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.273210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.273530 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.274039 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.274059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.275299 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.275303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.276427 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.276438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.277171 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.277176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.277412 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.278603 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.278619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.280115 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.282402 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.282408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.282422 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.282804 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.283274 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.283281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.283284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.283615 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.284218 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.284224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.284536 (+ 312us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.284716 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.284966 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260808 06:00:21.287964 22376 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43329
I20260808 06:00:21.288110 22376 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:21.288631 22376 heartbeater.cc:511] Master 127.13.229.126:43329 requested a full tablet report, sending...
I20260808 06:00:21.290356 22161 ts_manager.cc:194] Registered new tserver with Master: 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477)
I20260808 06:00:21.292321 22161 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:40917
WARNING: no policy specified for oryx/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:21.299432 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.289002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.65:37779)
0808 06:00:21.292078 (+ 3076us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.292091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.292208 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.292495 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.292498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.292691 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.292787 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.292792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.293766 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.293769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.294759 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.294771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.295275 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.295279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.295380 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.295790 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.295801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.296224 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.297869 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.297873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.297875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.298148 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.298494 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.298496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.298497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.298573 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.299048 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.299051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.299127 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.299205 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.299305 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3002,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260808 06:00:21.299682 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.289063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:37779 (local address 127.13.229.126:43329)
0808 06:00:21.289194 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.289198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.292163 (+ 2965us) server_negotiation.cc:407] Connection header received
0808 06:00:21.292299 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.292303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.292349 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.292443 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.292916 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.293607 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.294897 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.295155 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.295238 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.296346 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.296349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.296351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.296379 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.297667 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.298280 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.298284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.298286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.298394 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.298666 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.298672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.298677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.298941 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.299077 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.299289 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.299435 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
I20260808 06:00:21.314555 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43329
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:21.425886 22385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:21.426134 22385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:21.426174 22385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:21.430102 22385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:21.430152 22385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:21.430190 22385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:00:21.430289 22385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:21.435309 22385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:21.436321 22385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:21.437358 22385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:21.447570 22385 server_base.cc:1034] running on GCE node
W20260808 06:00:21.447527 22393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.447528 22390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.447713 22391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:21.448227 22385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:21.448772 22385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:21.449939 22385 hybrid_clock.cc:648] HybridClock initialized: now 1786168821449915 us; error 55 us; skew 500 ppm
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:21.452746 22385 init.cc:377] Logged in from keytab as oryx/127.13.229.66@KRBTEST.COM (short username oryx)
I20260808 06:00:21.453575 22385 webserver.cc:460] Webserver started at http://127.13.229.66:40065/ using document root <none> and password file <none>
I20260808 06:00:21.454134 22385 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:21.454223 22385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:21.454465 22385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:21.456698 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "e92ffe25ce5d476888c2507ea4ef3571"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "ae2dde8f71b9ee6eed0d2b1a9f950e1e"
server_key_iv: "dd6604e7d3e3d953d189387bcf8579c0"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.457271 22385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "e92ffe25ce5d476888c2507ea4ef3571"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "ae2dde8f71b9ee6eed0d2b1a9f950e1e"
server_key_iv: "dd6604e7d3e3d953d189387bcf8579c0"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.463016 22385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 06:00:21.466002 22401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.467571 22385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:21.467669 22385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "e92ffe25ce5d476888c2507ea4ef3571"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "ae2dde8f71b9ee6eed0d2b1a9f950e1e"
server_key_iv: "dd6604e7d3e3d953d189387bcf8579c0"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.467752 22385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:21.500910 22385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:21.503993 22385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:21.504148 22385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:21.505529 22385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:21.505577 22385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.505641 22385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:21.505671 22385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:21.516479 22385 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:33951
I20260808 06:00:21.516594 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:33951 every 8 connection(s)
I20260808 06:00:21.517472 22385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:21.520139 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.506416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:59523 (local address 127.13.229.126:43329)
0808 06:00:21.506568 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.506573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.507476 (+ 903us) server_negotiation.cc:407] Connection header received
0808 06:00:21.508651 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.508654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.508710 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.508801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.510432 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.511321 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.513250 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.513524 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.513607 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.516394 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.516398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.516401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.516429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.518261 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.518781 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.518784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.518785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.518866 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.519148 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.519150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.519151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.519331 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.519407 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.519856 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.520024 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 06:00:21.520816 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.506823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.66:59523)
0808 06:00:21.507301 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.507349 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.508420 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.509026 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.509037 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.509533 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.510249 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.510274 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.511471 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.511476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.513090 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.513105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.513701 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.513706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.513938 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.514550 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.514569 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.516191 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.518400 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.518405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.518419 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.518667 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.518946 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.518947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.518949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.519062 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.519421 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.519425 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.519630 (+ 205us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.519772 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.520071 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":144,"threads_started":1}
I20260808 06:00:21.521018 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22385
I20260808 06:00:21.521145 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:21.521482 14229 external_mini_cluster.cc:1428] Setting key 8407f4a55b93c444c7270130b5bf2434
I20260808 06:00:21.522147 22521 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43329
I20260808 06:00:21.522298 22521 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:21.524005 22521 heartbeater.cc:511] Master 127.13.229.126:43329 requested a full tablet report, sending...
I20260808 06:00:21.525358 22162 ts_manager.cc:194] Registered new tserver with Master: e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:21.526193 22162 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:59523
I20260808 06:00:21.531415 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.524008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:37175 (local address 127.13.229.126:43329)
0808 06:00:21.524126 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.524131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.524485 (+ 354us) server_negotiation.cc:407] Connection header received
0808 06:00:21.524599 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.524603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.524646 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.524747 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.525219 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.525798 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.526754 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.527148 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.527282 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.528438 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.528442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.528444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.528468 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.529844 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.530237 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.530241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.530243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.530315 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.530576 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.530580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.530581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.530789 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.530913 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.531160 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.531264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:00:21.531450 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.524103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.66:37175)
0808 06:00:21.524391 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.524406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.524524 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.524762 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.524764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.524991 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.525129 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.525137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.525930 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.525933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.526633 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.526642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.527264 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.527266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.527332 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.527787 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.527800 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.528245 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.529965 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.529968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.529970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.530144 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.530406 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.530410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.530412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.530492 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.530907 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.530910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.531003 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.531108 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.531236 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for oryx/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
I20260808 06:00:21.550454 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:43329
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:21.661068 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:21.661311 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:21.661378 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:21.665338 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:21.665387 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:21.665455 22531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 06:00:21.665513 22531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:21.670503 22531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.229.126:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:21.671393 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:21.672439 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:21.681708 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.681715 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:21.681730 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:21.681861 22531 server_base.cc:1034] running on GCE node
I20260808 06:00:21.682366 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:21.682996 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:21.684175 22531 hybrid_clock.cc:648] HybridClock initialized: now 1786168821684150 us; error 39 us; skew 500 ppm
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:21.687126 22531 init.cc:377] Logged in from keytab as oryx/127.13.229.67@KRBTEST.COM (short username oryx)
I20260808 06:00:21.688064 22531 webserver.cc:460] Webserver started at http://127.13.229.67:38787/ using document root <none> and password file <none>
I20260808 06:00:21.688558 22531 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:21.688612 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:21.688764 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:21.690634 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "c767ed77cd5744969eb50284181dd95d"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "f66b317e6e7ca29a8c3ee45ca48b3a0d"
server_key_iv: "46361e42dad4ceeeea1461dd3f9e7823"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.691136 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "c767ed77cd5744969eb50284181dd95d"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "f66b317e6e7ca29a8c3ee45ca48b3a0d"
server_key_iv: "46361e42dad4ceeeea1461dd3f9e7823"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.694691 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 06:00:21.696973 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.698114 22531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:21.698236 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "c767ed77cd5744969eb50284181dd95d"
format_stamp: "Formatted at 2026-08-08 06:00:21 on dist-test-slave-zkpd"
server_key: "f66b317e6e7ca29a8c3ee45ca48b3a0d"
server_key_iv: "46361e42dad4ceeeea1461dd3f9e7823"
server_key_version: "encryptionkey@0"
I20260808 06:00:21.698336 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:21.717800 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:21.721011 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:21.721208 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:21.722714 22531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:21.722759 22531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:21.722797 22531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:21.722852 22531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:21.733942 22531 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41407
I20260808 06:00:21.734041 22666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41407 every 8 connection(s)
I20260808 06:00:21.734949 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:21.736660 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22531
I20260808 06:00:21.736805 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:21.737107 14229 external_mini_cluster.cc:1428] Setting key dc411b54445688b0a614ce768ea11027
I20260808 06:00:21.737816 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.723841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:42473 (local address 127.13.229.126:43329)
0808 06:00:21.724008 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.724013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.724912 (+ 899us) server_negotiation.cc:407] Connection header received
0808 06:00:21.726292 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.726297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.726372 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.726493 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.728205 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.728851 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.730301 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.730566 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.730694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.733579 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.733583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.733586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.733646 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.735277 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.735978 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.735982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.735984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.736080 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.736483 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.736487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.736489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.736689 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.736824 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.737482 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.737618 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 06:00:21.738653 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.724215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.67:42473)
0808 06:00:21.724731 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.724787 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.726030 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.726721 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.726730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.727259 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.728022 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.728044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.729011 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.729015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.730155 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.730167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.730738 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.730743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.730926 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.731488 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.731505 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.733387 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.735408 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.735414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.735430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.735823 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.736190 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.736194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.736196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.736349 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.736906 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.736913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.737186 (+ 273us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.737411 (+ 225us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.737743 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260808 06:00:21.740016 22667 heartbeater.cc:347] Connected to a master server at 127.13.229.126:43329
I20260808 06:00:21.740172 22667 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:21.740768 22667 heartbeater.cc:511] Master 127.13.229.126:43329 requested a full tablet report, sending...
I20260808 06:00:21.741771 22161 ts_manager.cc:194] Registered new tserver with Master: c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407)
I20260808 06:00:21.742506 22161 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:42473
I20260808 06:00:21.749310 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.742250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.13.229.67:53277)
0808 06:00:21.742552 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.742568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.742663 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.743081 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.743084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.743291 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:21.743464 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.743475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.744380 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.744383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.744976 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.744985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.745428 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.745430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.745501 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.746048 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:21.746059 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:21.746336 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:21.748020 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.748022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.748024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.748204 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.748470 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:21.748472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:21.748474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:21.748544 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:21.748911 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:21.748913 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:21.748983 (+ 70us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.749062 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.749159 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
I20260808 06:00:21.749472 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.742309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:53277 (local address 127.13.229.126:43329)
0808 06:00:21.742451 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.742455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.742673 (+ 218us) server_negotiation.cc:407] Connection header received
0808 06:00:21.742885 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.742888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.742940 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.743037 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.743588 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.744240 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.745092 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.745327 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.745414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.746468 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.746471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.746473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.746495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.747858 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.748300 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.748303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.748304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.748384 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.748624 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.748627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.748628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.748824 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.748908 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.749173 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.749287 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:21.751302 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:00:21.776031 22212 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:21.769829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.13.229.126:43329)
0808 06:00:21.769965 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.769969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.769981 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:00:21.770016 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.770019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.770057 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.770150 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.770924 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.771438 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.772352 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.772570 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.772677 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.775857 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37818: BlockingRecv error: recv got EOF from 127.0.0.1:37818 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 06:00:21 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:21.786000 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.777413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.13.229.126:43329)
0808 06:00:21.777577 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.777582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.777782 (+ 200us) server_negotiation.cc:407] Connection header received
0808 06:00:21.777945 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.777948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.778001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.778104 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:21.778785 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.779348 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.780332 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.780572 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.780688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.783088 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:21.783099 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:21.783102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:21.783124 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:21.784356 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.784815 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.784818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.784819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.784893 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:21.785171 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:21.785175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:21.785177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:21.785349 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:21.785488 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.785720 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.785856 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 06:00:21.788817 22162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:21.791776 22162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:21.806034 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.799839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41407 (local address 127.0.0.1:34224)
0808 06:00:21.800579 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.800670 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.800857 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.802203 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.802206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.802238 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:21.802327 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.802333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803559 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.803562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.804689 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.804699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.805502 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.805508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.805618 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.805659 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.805769 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.805884 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260808 06:00:21.806046 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.800029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.13.229.67:41407)
0808 06:00:21.801739 (+ 1710us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.801747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.801766 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:00:21.801829 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.801832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.802028 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.802240 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:21.802581 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803400 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.804822 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.805384 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.805492 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.805654 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.805807 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.805902 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1625,"thread_start_us":123,"threads_started":1}
I20260808 06:00:21.806922 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.800573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:38477 (local address 127.0.0.1:55782)
0808 06:00:21.801433 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.801446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.801538 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.801973 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.801979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.802000 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:21.802234 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.802244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803761 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.803767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.805244 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.805254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.806453 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.806457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.806558 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.806585 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.806654 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.806737 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":108,"thread_start_us":93,"threads_started":1}
I20260808 06:00:21.806923 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.800117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33951 (local address 127.0.0.1:45266)
0808 06:00:21.801077 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.801091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.801171 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.801978 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.801981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.802017 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:21.802464 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.802473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803841 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.803844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.805244 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.805254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.806518 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.806521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.806621 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.806655 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.806697 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.806745 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":408,"spinlock_wait_cycles":4992,"thread_start_us":106,"threads_started":1}
I20260808 06:00:21.807333 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.800881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.13.229.65:38477)
0808 06:00:21.801179 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.801185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.801540 (+ 355us) server_negotiation.cc:407] Connection header received
0808 06:00:21.801613 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.801618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.801810 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.801969 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:21.802382 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803625 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.805384 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.806326 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.806437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.806565 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.807102 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.807201 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":91,"threads_started":1}
I20260808 06:00:21.807655 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.800928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.13.229.66:33951)
0808 06:00:21.801462 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.801480 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.801517 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:00:21.801601 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.801606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.801811 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.802233 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:21.802597 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.803722 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.805387 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.806369 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.807209 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.807381 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.807435 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.807525 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":114,"threads_started":1}
I20260808 06:00:21.809244 22598 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0178b1c5144478514b644cc7da0af (DEFAULT_TABLE table=test-table [id=258e9e4e233e4a67b111d49c435f54a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:21.809872 22454 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0178b1c5144478514b644cc7da0af (DEFAULT_TABLE table=test-table [id=258e9e4e233e4a67b111d49c435f54a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:21.810395 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0178b1c5144478514b644cc7da0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:21.810225 22305 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0178b1c5144478514b644cc7da0af (DEFAULT_TABLE table=test-table [id=258e9e4e233e4a67b111d49c435f54a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:21.810642 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0178b1c5144478514b644cc7da0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:21.811180 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0178b1c5144478514b644cc7da0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:21.817078 22695 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Bootstrap starting.
I20260808 06:00:21.818370 22697 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Bootstrap starting.
I20260808 06:00:21.818820 22696 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Bootstrap starting.
I20260808 06:00:21.819309 22695 tablet_bootstrap.cc:654] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:21.820252 22695 log.cc:824] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:21.820554 22697 tablet_bootstrap.cc:654] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:21.820739 22696 tablet_bootstrap.cc:654] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:21.821424 22697 log.cc:824] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:21.821568 22696 log.cc:824] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:21.822522 22695 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: No bootstrap required, opened a new log
I20260808 06:00:21.822705 22695 ts_tablet_manager.cc:1397] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:00:21.823508 22696 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: No bootstrap required, opened a new log
I20260808 06:00:21.823691 22696 ts_tablet_manager.cc:1397] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 06:00:21.826665 22695 raft_consensus.cc:348] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.826942 22695 raft_consensus.cc:374] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:21.826999 22695 raft_consensus.cc:729] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Initialized, Role: FOLLOWER
I20260808 06:00:21.827445 22695 consensus_queue.cc:260] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.827512 22696 raft_consensus.cc:348] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.827751 22696 raft_consensus.cc:374] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:21.827812 22696 raft_consensus.cc:729] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Initialized, Role: FOLLOWER
I20260808 06:00:21.828074 22697 tablet_bootstrap.cc:492] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: No bootstrap required, opened a new log
I20260808 06:00:21.828271 22697 ts_tablet_manager.cc:1397] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260808 06:00:21.828240 22696 consensus_queue.cc:260] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.828500 22695 ts_tablet_manager.cc:1428] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:00:21.828892 22521 heartbeater.cc:503] Master 127.13.229.126:43329 was elected leader, sending a full tablet report...
I20260808 06:00:21.828943 22667 heartbeater.cc:503] Master 127.13.229.126:43329 was elected leader, sending a full tablet report...
I20260808 06:00:21.829186 22696 ts_tablet_manager.cc:1428] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:00:21.833142 22697 raft_consensus.cc:348] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.833400 22697 raft_consensus.cc:374] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:21.833463 22697 raft_consensus.cc:729] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abff3f82b3946feabc50370a0c01450, State: Initialized, Role: FOLLOWER
I20260808 06:00:21.833855 22697 consensus_queue.cc:260] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.834429 22376 heartbeater.cc:503] Master 127.13.229.126:43329 was elected leader, sending a full tablet report...
I20260808 06:00:21.834583 22697 ts_tablet_manager.cc:1428] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:00:21.966427 22703 raft_consensus.cc:482] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:21.966671 22703 raft_consensus.cc:504] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.967847 22703 leader_election.cc:288] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951), 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477)
I20260808 06:00:21.972033 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.968182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:38477 (local address 127.13.229.67:37715)
0808 06:00:21.968327 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.968343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.968451 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.968757 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.968760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.968805 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:21.968933 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.968938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.969739 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.969741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.970842 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.970853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.971574 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.971577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.971659 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.971680 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.971739 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.971852 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":28}
I20260808 06:00:21.972043 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.968292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:37715 (local address 127.13.229.65:38477)
0808 06:00:21.968436 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.968440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.968456 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:21.968588 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.968591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.968644 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.968712 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:21.969076 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.969628 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.970971 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.971445 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.971584 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.971639 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.971776 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.971880 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 06:00:21.972757 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.968195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33951 (local address 127.13.229.67:57213)
0808 06:00:21.968646 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:21.968656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:21.968769 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:21.968999 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:21.969003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:21.969025 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:21.969484 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.969490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.970259 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.970263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.971254 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:21.971262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.972308 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.972313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:21.972432 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.972461 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 06:00:21.972539 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 06:00:21.972618 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":132,"thread_start_us":114,"threads_started":1}
I20260808 06:00:21.972815 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.968346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:57213 (local address 127.13.229.66:33951)
0808 06:00:21.968489 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.968494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.968768 (+ 274us) server_negotiation.cc:407] Connection header received
0808 06:00:21.968861 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.968864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.968905 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.968964 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:21.969605 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.970148 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.971378 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.972014 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.972137 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.972192 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.972626 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.972701 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:00:21.972805 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0178b1c5144478514b644cc7da0af" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450" is_pre_election: true
I20260808 06:00:21.973097 22327 raft_consensus.cc:2454] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:21.973534 22472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0178b1c5144478514b644cc7da0af" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571" is_pre_election: true
I20260808 06:00:21.973582 22549 leader_election.cc:302] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8abff3f82b3946feabc50370a0c01450, c767ed77cd5744969eb50284181dd95d; no voters:
I20260808 06:00:21.973822 22472 raft_consensus.cc:2454] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:21.973922 22703 raft_consensus.cc:2781] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:21.974015 22703 raft_consensus.cc:482] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:21.974069 22703 raft_consensus.cc:3037] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:21.975132 22703 raft_consensus.cc:504] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.975463 22703 leader_election.cc:288] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Requested vote from peers e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951), 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477)
I20260808 06:00:21.975992 22472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0178b1c5144478514b644cc7da0af" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571"
I20260808 06:00:21.976007 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0178b1c5144478514b644cc7da0af" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450"
I20260808 06:00:21.976122 22472 raft_consensus.cc:3037] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:21.976133 22327 raft_consensus.cc:3037] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:21.977246 22472 raft_consensus.cc:2454] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:21.977458 22327 raft_consensus.cc:2454] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:21.977581 22547 leader_election.cc:302] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c767ed77cd5744969eb50284181dd95d, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:21.977787 22703 raft_consensus.cc:2781] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:21.978022 22703 raft_consensus.cc:686] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Becoming Leader. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Running, Role: LEADER
I20260808 06:00:21.978327 22703 consensus_queue.cc:237] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } }
I20260808 06:00:21.981637 22161 catalog_manager.cc:5495] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d reported cstate change: term changed from 0 to 1, leader changed from <none> to c767ed77cd5744969eb50284181dd95d (127.13.229.67). New cstate: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } health_report { overall_health: UNKNOWN } } }
W20260808 06:00:21.986388 22668 tablet.cc:2369] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:00:21.997059 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:21.993913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.13.229.67:41407)
0808 06:00:21.994024 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:21.994028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:21.994108 (+ 80us) server_negotiation.cc:407] Connection header received
0808 06:00:21.994213 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:21.994215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:21.994260 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:21.994340 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:21.994548 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.995023 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.995837 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:21.996104 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:21.996184 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:21.996430 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:21.996789 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:21.996883 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:21.996931 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 06:00:21.996975 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:00:22.002583 22327 raft_consensus.cc:1261] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.002656 22472 raft_consensus.cc:1261] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.003307 22706 consensus_queue.cc:1035] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.003507 22703 consensus_queue.cc:1035] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.013106 22710 mvcc.cc:205] Tried to move back new op lower bound from 7316147494915686400 to 7316147494825979904. Current Snapshot: MvccSnapshot[applied={T|T < 7316147494915686400}]
I20260808 06:00:22.017671 22162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.017827 22162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.020337 22162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Sending DeleteTablet for 3 replicas of tablet d0b0178b1c5144478514b644cc7da0af
I20260808 06:00:22.021124 22305 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0178b1c5144478514b644cc7da0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:55782
I20260808 06:00:22.021188 22598 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0178b1c5144478514b644cc7da0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:34224
I20260808 06:00:22.021624 22454 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0178b1c5144478514b644cc7da0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:45266
I20260808 06:00:22.021667 22162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 06:00:22.022006 22721 tablet_replica.cc:330] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: stopping tablet replica
W20260808 06:00:22.022105 22162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:22.022342 22721 raft_consensus.cc:2229] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.022643 22721 raft_consensus.cc:2258] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.023456 22720 tablet_replica.cc:330] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: stopping tablet replica
I20260808 06:00:22.023751 22720 raft_consensus.cc:2229] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.024053 22720 raft_consensus.cc:2258] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.024117 22721 ts_tablet_manager.cc:1905] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.025391 22720 ts_tablet_manager.cc:1905] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.026882 22721 ts_tablet_manager.cc:1918] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.026983 22721 log.cc:1196] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/d0b0178b1c5144478514b644cc7da0af
I20260808 06:00:22.027302 22721 ts_tablet_manager.cc:1939] T d0b0178b1c5144478514b644cc7da0af P e92ffe25ce5d476888c2507ea4ef3571: Deleting consensus metadata
I20260808 06:00:22.028146 22720 ts_tablet_manager.cc:1918] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.028249 22720 log.cc:1196] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/d0b0178b1c5144478514b644cc7da0af
I20260808 06:00:22.028556 22720 ts_tablet_manager.cc:1939] T d0b0178b1c5144478514b644cc7da0af P 8abff3f82b3946feabc50370a0c01450: Deleting consensus metadata
I20260808 06:00:22.028556 22305 tablet_service.cc:1467] Processing CreateTablet for tablet f057e5b5a537467281f78f1e2df26778 (DEFAULT_TABLE table=test-table [id=16ccead4690a49a2815ba7f7b762fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.028887 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f057e5b5a537467281f78f1e2df26778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.028890 22146 catalog_manager.cc:4841] TS e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951): tablet d0b0178b1c5144478514b644cc7da0af (table test-table [id=258e9e4e233e4a67b111d49c435f54a0]) successfully deleted
I20260808 06:00:22.029767 22719 tablet_replica.cc:330] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: stopping tablet replica
I20260808 06:00:22.029896 22148 catalog_manager.cc:4841] TS 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477): tablet d0b0178b1c5144478514b644cc7da0af (table test-table [id=258e9e4e233e4a67b111d49c435f54a0]) successfully deleted
I20260808 06:00:22.030081 22719 raft_consensus.cc:2229] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:22.030406 22719 raft_consensus.cc:2258] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.030895 22697 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Bootstrap starting.
I20260808 06:00:22.031065 22598 tablet_service.cc:1467] Processing CreateTablet for tablet f057e5b5a537467281f78f1e2df26778 (DEFAULT_TABLE table=test-table [id=16ccead4690a49a2815ba7f7b762fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.031297 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f057e5b5a537467281f78f1e2df26778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.031797 22697 tablet_bootstrap.cc:654] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.032315 22454 tablet_service.cc:1467] Processing CreateTablet for tablet f057e5b5a537467281f78f1e2df26778 (DEFAULT_TABLE table=test-table [id=16ccead4690a49a2815ba7f7b762fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.032539 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f057e5b5a537467281f78f1e2df26778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.033181 22719 ts_tablet_manager.cc:1905] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.034724 22695 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Bootstrap starting.
I20260808 06:00:22.035661 22695 tablet_bootstrap.cc:654] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.035763 22719 ts_tablet_manager.cc:1918] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.035868 22719 log.cc:1196] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/d0b0178b1c5144478514b644cc7da0af
I20260808 06:00:22.036216 22719 ts_tablet_manager.cc:1939] T d0b0178b1c5144478514b644cc7da0af P c767ed77cd5744969eb50284181dd95d: Deleting consensus metadata
I20260808 06:00:22.036546 22696 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Bootstrap starting.
I20260808 06:00:22.036983 22697 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: No bootstrap required, opened a new log
I20260808 06:00:22.037058 22697 ts_tablet_manager.cc:1397] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260808 06:00:22.037485 22696 tablet_bootstrap.cc:654] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.037492 22697 raft_consensus.cc:348] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.037596 22697 raft_consensus.cc:374] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.037628 22697 raft_consensus.cc:729] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abff3f82b3946feabc50370a0c01450, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.037742 22697 consensus_queue.cc:260] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.037854 22148 catalog_manager.cc:4841] TS c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407): tablet d0b0178b1c5144478514b644cc7da0af (table test-table [id=258e9e4e233e4a67b111d49c435f54a0]) successfully deleted
I20260808 06:00:22.037983 22697 ts_tablet_manager.cc:1428] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.039012 22696 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: No bootstrap required, opened a new log
I20260808 06:00:22.039117 22696 ts_tablet_manager.cc:1397] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.039635 22696 raft_consensus.cc:348] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.039798 22696 raft_consensus.cc:374] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.039841 22696 raft_consensus.cc:729] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.039997 22696 consensus_queue.cc:260] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.040297 22696 ts_tablet_manager.cc:1428] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.041141 22695 tablet_bootstrap.cc:492] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: No bootstrap required, opened a new log
I20260808 06:00:22.041247 22695 ts_tablet_manager.cc:1397] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 06:00:22.041672 22695 raft_consensus.cc:348] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.041785 22695 raft_consensus.cc:374] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.041828 22695 raft_consensus.cc:729] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.041970 22695 consensus_queue.cc:260] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.042254 22695 ts_tablet_manager.cc:1428] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 06:00:22.068111 22716 raft_consensus.cc:482] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:22.068248 22716 raft_consensus.cc:504] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.068701 22716 leader_election.cc:288] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.069106 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f057e5b5a537467281f78f1e2df26778" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571" is_pre_election: true
I20260808 06:00:22.069115 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f057e5b5a537467281f78f1e2df26778" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450" is_pre_election: true
I20260808 06:00:22.069263 22327 raft_consensus.cc:2454] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.069265 22474 raft_consensus.cc:2454] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.069581 22547 leader_election.cc:302] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c767ed77cd5744969eb50284181dd95d, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:22.069770 22716 raft_consensus.cc:2781] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:22.069831 22716 raft_consensus.cc:482] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:22.069877 22716 raft_consensus.cc:3037] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.071051 22716 raft_consensus.cc:504] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.071406 22716 leader_election.cc:288] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Requested vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.071748 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f057e5b5a537467281f78f1e2df26778" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450"
I20260808 06:00:22.071746 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f057e5b5a537467281f78f1e2df26778" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571"
I20260808 06:00:22.071877 22474 raft_consensus.cc:3037] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.071877 22327 raft_consensus.cc:3037] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.073192 22474 raft_consensus.cc:2454] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.073457 22547 leader_election.cc:302] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c767ed77cd5744969eb50284181dd95d, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:22.073705 22327 raft_consensus.cc:2454] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.074045 22716 raft_consensus.cc:2781] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:22.074146 22716 raft_consensus.cc:686] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Becoming Leader. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Running, Role: LEADER
I20260808 06:00:22.074282 22716 consensus_queue.cc:237] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.075877 22161 catalog_manager.cc:5495] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d reported cstate change: term changed from 0 to 1, leader changed from <none> to c767ed77cd5744969eb50284181dd95d (127.13.229.67). New cstate: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } health_report { overall_health: HEALTHY } } }
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168820, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.126@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:22.117384 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.101703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:43329 (local address 127.0.0.1:37838)
0808 06:00:22.102005 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:22.102020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:22.102175 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:22.102507 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:22.102510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:22.103041 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:22.103193 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.103202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.104071 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.104075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.104978 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.104988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.105401 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.105405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.105510 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.112175 (+ 6665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:22.112191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:22.113907 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:22.115681 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:22.115691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:22.115697 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:22.116006 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:22.116281 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:22.116285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:22.116286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:22.116372 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:22.116758 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:22.116765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:22.116893 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 06:00:22.117013 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 06:00:22.117182 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":128,"threads_started":1}
I20260808 06:00:22.117383 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.101777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.13.229.126:43329)
0808 06:00:22.101902 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.101906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.102124 (+ 218us) server_negotiation.cc:407] Connection header received
0808 06:00:22.102302 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.102320 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.102370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.102472 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:22.103368 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.103963 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.105093 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.105300 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.105374 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.114068 (+ 8694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:22.114071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:22.114073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:22.114094 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:22.115552 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:22.116102 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:22.116108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:22.116110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:22.116185 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:22.116462 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:22.116467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:22.116469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:22.116651 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:22.116758 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.117062 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.117261 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":41}
I20260808 06:00:22.121548 22161 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:37838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 06:00:22.122457 22161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:22.127769 22305 tablet_service.cc:1467] Processing CreateTablet for tablet 53ee6b3ea5a14153894749b3673c88a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=806e0b42e8f84d2a9e33ef3ba3813175]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:00:22.127770 22454 tablet_service.cc:1467] Processing CreateTablet for tablet 53ee6b3ea5a14153894749b3673c88a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=806e0b42e8f84d2a9e33ef3ba3813175]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:00:22.127767 22598 tablet_service.cc:1467] Processing CreateTablet for tablet 53ee6b3ea5a14153894749b3673c88a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=806e0b42e8f84d2a9e33ef3ba3813175]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 06:00:22.128197 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ee6b3ea5a14153894749b3673c88a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.128201 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ee6b3ea5a14153894749b3673c88a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.128201 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ee6b3ea5a14153894749b3673c88a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.130978 22696 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: Bootstrap starting.
I20260808 06:00:22.131086 22697 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: Bootstrap starting.
I20260808 06:00:22.131860 22696 tablet_bootstrap.cc:654] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.132164 22697 tablet_bootstrap.cc:654] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.132678 22695 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: Bootstrap starting.
I20260808 06:00:22.133582 22697 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: No bootstrap required, opened a new log
I20260808 06:00:22.133607 22696 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: No bootstrap required, opened a new log
I20260808 06:00:22.133657 22697 ts_tablet_manager.cc:1397] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.133690 22696 ts_tablet_manager.cc:1397] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.133817 22695 tablet_bootstrap.cc:654] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.134467 22696 raft_consensus.cc:348] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.134693 22696 raft_consensus.cc:374] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.134652 22697 raft_consensus.cc:348] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.134738 22696 raft_consensus.cc:729] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.134788 22697 raft_consensus.cc:374] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.134822 22697 raft_consensus.cc:729] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abff3f82b3946feabc50370a0c01450, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.134959 22697 consensus_queue.cc:260] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.135025 22696 consensus_queue.cc:260] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.135241 22697 ts_tablet_manager.cc:1428] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:22.135494 22704 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.135632 22704 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.135619 22716 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.136019 22716 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.138346 22696 ts_tablet_manager.cc:1428] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 06:00:22.140311 22695 tablet_bootstrap.cc:492] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: No bootstrap required, opened a new log
I20260808 06:00:22.140412 22695 ts_tablet_manager.cc:1397] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260808 06:00:22.140928 22695 raft_consensus.cc:348] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.141075 22695 raft_consensus.cc:374] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.141120 22695 raft_consensus.cc:729] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.141513 22695 consensus_queue.cc:260] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.141782 22702 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.141937 22702 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.142735 22695 ts_tablet_manager.cc:1428] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260808 06:00:22.269541 22522 tablet.cc:2369] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:00:22.283186 22377 tablet.cc:2369] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:00:22.335686 22716 raft_consensus.cc:482] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:22.335872 22716 raft_consensus.cc:504] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.336397 22716 leader_election.cc:288] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.336881 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ee6b3ea5a14153894749b3673c88a9" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450" is_pre_election: true
I20260808 06:00:22.336881 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ee6b3ea5a14153894749b3673c88a9" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571" is_pre_election: true
I20260808 06:00:22.337039 22474 raft_consensus.cc:2454] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.337039 22327 raft_consensus.cc:2454] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.337400 22549 leader_election.cc:302] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8abff3f82b3946feabc50370a0c01450, c767ed77cd5744969eb50284181dd95d; no voters:
I20260808 06:00:22.337590 22716 raft_consensus.cc:2781] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:22.337656 22716 raft_consensus.cc:482] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:22.337683 22716 raft_consensus.cc:3037] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.338646 22716 raft_consensus.cc:504] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.338980 22716 leader_election.cc:288] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Requested vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.339259 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ee6b3ea5a14153894749b3673c88a9" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450"
I20260808 06:00:22.339422 22327 raft_consensus.cc:3037] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.339394 22474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ee6b3ea5a14153894749b3673c88a9" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571"
I20260808 06:00:22.339510 22474 raft_consensus.cc:3037] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.340651 22474 raft_consensus.cc:2454] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.340647 22327 raft_consensus.cc:2454] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.340946 22549 leader_election.cc:302] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8abff3f82b3946feabc50370a0c01450, c767ed77cd5744969eb50284181dd95d; no voters:
I20260808 06:00:22.341123 22716 raft_consensus.cc:2781] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:22.341253 22716 raft_consensus.cc:686] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Becoming Leader. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Running, Role: LEADER
I20260808 06:00:22.341424 22716 consensus_queue.cc:237] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.342033 22706 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: TxnStatusTablet state changed. Reason: New leader c767ed77cd5744969eb50284181dd95d. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.342149 22706 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:00:22.342371 22737 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 06:00:22.342545 22737 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 06:00:22.343137 22161 catalog_manager.cc:5495] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d reported cstate change: term changed from 0 to 1, leader changed from <none> to c767ed77cd5744969eb50284181dd95d (127.13.229.67). New cstate: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } health_report { overall_health: HEALTHY } } }
I20260808 06:00:22.379429 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.375942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41407 (local address 127.0.0.1:34246)
0808 06:00:22.376092 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:22.376110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:22.376247 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:22.376540 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:22.376543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:22.376549 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:00:22.376641 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.376645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.377626 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.377629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.378228 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.378236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.378783 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.378785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.378849 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.378906 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.379188 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.379199 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 06:00:22.379249 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 06:00:22.379313 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 06:00:22.379444 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.376028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.13.229.67:41407)
0808 06:00:22.376144 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.376148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.376162 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:22.376370 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.376373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.376428 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.376510 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:22.376740 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.377521 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.378339 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.378676 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.378778 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.379027 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.379108 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.379179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.379259 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.379315 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:00:22.381995 22474 raft_consensus.cc:1261] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.382134 22327 raft_consensus.cc:1261] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.382457 22706 consensus_queue.cc:1035] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.382594 22716 consensus_queue.cc:1035] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.384282 22704 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: TxnStatusTablet state changed. Reason: New leader c767ed77cd5744969eb50284181dd95d. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.384394 22704 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.384892 22702 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: TxnStatusTablet state changed. Reason: New leader c767ed77cd5744969eb50284181dd95d. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.385004 22702 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.386641 22716 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.386752 22716 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:00:22.387017 22706 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.387108 22706 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P c767ed77cd5744969eb50284181dd95d: This TxnStatusTablet replica's current role is: LEADER
I20260808 06:00:22.387871 22711 mvcc.cc:205] Tried to move back new op lower bound from 7316147496472694784 to 7316147496311762944. Current Snapshot: MvccSnapshot[applied={T|T < 7316147496472694784}]
I20260808 06:00:22.388017 22704 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.388119 22704 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P 8abff3f82b3946feabc50370a0c01450: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.388391 22709 mvcc.cc:205] Tried to move back new op lower bound from 7316147496472694784 to 7316147496311762944. Current Snapshot: MvccSnapshot[applied={T|T < 7316147496472694784 or (T in {7316147496472694784})}]
I20260808 06:00:22.390015 22702 tablet_replica.cc:439] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } } }
I20260808 06:00:22.390115 22702 tablet_replica.cc:442] T 53ee6b3ea5a14153894749b3673c88a9 P e92ffe25ce5d476888c2507ea4ef3571: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 06:00:22.401507 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.398039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54183 (local address 127.13.229.67:41407)
0808 06:00:22.398154 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.398159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.398312 (+ 153us) server_negotiation.cc:407] Connection header received
0808 06:00:22.398465 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.398471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.398516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.398592 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:22.398859 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.399397 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.400416 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.400687 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.400792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.401094 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.401154 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.401229 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.401316 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.401367 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":41}
I20260808 06:00:22.401507 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.397965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41407 (local address 127.13.229.67:54183)
0808 06:00:22.398236 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:22.398247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:22.398362 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:22.398600 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:22.398602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:22.398609 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 06:00:22.398752 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.398758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.399500 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.399505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.400286 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.400296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.400792 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.400795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.400895 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.400936 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.401243 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.401250 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 06:00:22.401309 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:00:22.401384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260808 06:00:22.407413 22327 raft_consensus.cc:1261] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.407548 22472 raft_consensus.cc:1261] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.408155 22706 consensus_queue.cc:1035] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.408486 22703 consensus_queue.cc:1035] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.448086 22161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.448266 22161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.450315 22161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Sending DeleteTablet for 3 replicas of tablet f057e5b5a537467281f78f1e2df26778
I20260808 06:00:22.450861 22305 tablet_service.cc:1514] Processing DeleteTablet for tablet f057e5b5a537467281f78f1e2df26778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:55782
I20260808 06:00:22.451073 22720 tablet_replica.cc:330] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: stopping tablet replica
I20260808 06:00:22.451059 22454 tablet_service.cc:1514] Processing DeleteTablet for tablet f057e5b5a537467281f78f1e2df26778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:45266
I20260808 06:00:22.451211 22720 raft_consensus.cc:2229] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.451232 22721 tablet_replica.cc:330] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: stopping tablet replica
I20260808 06:00:22.451351 22720 raft_consensus.cc:2258] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.451368 22721 raft_consensus.cc:2229] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.451496 22721 raft_consensus.cc:2258] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.451663 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet f057e5b5a537467281f78f1e2df26778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:34224
I20260808 06:00:22.451838 22719 tablet_replica.cc:330] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: stopping tablet replica
I20260808 06:00:22.452023 22719 raft_consensus.cc:2229] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:22.452219 22719 raft_consensus.cc:2258] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.452500 22721 ts_tablet_manager.cc:1905] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.452823 22720 ts_tablet_manager.cc:1905] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.455013 22720 ts_tablet_manager.cc:1918] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.455067 22720 log.cc:1196] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/f057e5b5a537467281f78f1e2df26778
I20260808 06:00:22.455255 22720 ts_tablet_manager.cc:1939] T f057e5b5a537467281f78f1e2df26778 P 8abff3f82b3946feabc50370a0c01450: Deleting consensus metadata
I20260808 06:00:22.455425 22721 ts_tablet_manager.cc:1918] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.455492 22721 log.cc:1196] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/f057e5b5a537467281f78f1e2df26778
I20260808 06:00:22.455731 22721 ts_tablet_manager.cc:1939] T f057e5b5a537467281f78f1e2df26778 P e92ffe25ce5d476888c2507ea4ef3571: Deleting consensus metadata
I20260808 06:00:22.456013 22148 catalog_manager.cc:4841] TS 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477): tablet f057e5b5a537467281f78f1e2df26778 (table test-table [id=16ccead4690a49a2815ba7f7b762fba5]) successfully deleted
I20260808 06:00:22.456538 22719 ts_tablet_manager.cc:1905] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.456934 22146 catalog_manager.cc:4841] TS e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951): tablet f057e5b5a537467281f78f1e2df26778 (table test-table [id=16ccead4690a49a2815ba7f7b762fba5]) successfully deleted
I20260808 06:00:22.458822 22719 ts_tablet_manager.cc:1918] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.458896 22719 log.cc:1196] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/f057e5b5a537467281f78f1e2df26778
I20260808 06:00:22.459172 22719 ts_tablet_manager.cc:1939] T f057e5b5a537467281f78f1e2df26778 P c767ed77cd5744969eb50284181dd95d: Deleting consensus metadata
I20260808 06:00:22.459831 22148 catalog_manager.cc:4841] TS c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407): tablet f057e5b5a537467281f78f1e2df26778 (table test-table [id=16ccead4690a49a2815ba7f7b762fba5]) successfully deleted
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:00:22.488252 22212 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:22.481375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.13.229.126:43329)
0808 06:00:22.481542 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.481548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.481667 (+ 119us) server_negotiation.cc:407] Connection header received
0808 06:00:22.481769 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.481773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.481834 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.481943 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:22.482671 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.483478 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.484389 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.484723 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.484838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.488115 (+ 3277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37848: BlockingRecv error: recv got EOF from 127.0.0.1:37848 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM
I20260808 06:00:22.497798 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.489745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.13.229.126:43329)
0808 06:00:22.489847 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.489851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.489990 (+ 139us) server_negotiation.cc:407] Connection header received
0808 06:00:22.490104 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.490107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.490145 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.490235 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:22.490802 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.491290 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.492138 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.492359 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.492476 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.494666 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:22.494675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:22.494677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:22.494699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:22.496147 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:22.496559 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:22.496562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:22.496564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:22.496645 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:22.496935 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:22.496939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:22.496941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:22.497111 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:22.497250 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.497481 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.497605 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 06:00:22.500511 22161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:22.500952 22161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:22.505897 22305 tablet_service.cc:1467] Processing CreateTablet for tablet 57132eac2a704ba2900187fa79c15d35 (DEFAULT_TABLE table=test-table [id=384d5186da584b5187ec1db00d11e079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.506009 22597 tablet_service.cc:1467] Processing CreateTablet for tablet 57132eac2a704ba2900187fa79c15d35 (DEFAULT_TABLE table=test-table [id=384d5186da584b5187ec1db00d11e079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.506273 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57132eac2a704ba2900187fa79c15d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.506321 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57132eac2a704ba2900187fa79c15d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.507074 22454 tablet_service.cc:1467] Processing CreateTablet for tablet 57132eac2a704ba2900187fa79c15d35 (DEFAULT_TABLE table=test-table [id=384d5186da584b5187ec1db00d11e079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.507375 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57132eac2a704ba2900187fa79c15d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.508415 22697 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Bootstrap starting.
I20260808 06:00:22.508872 22696 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Bootstrap starting.
I20260808 06:00:22.509399 22697 tablet_bootstrap.cc:654] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.509672 22696 tablet_bootstrap.cc:654] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.509759 22695 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Bootstrap starting.
I20260808 06:00:22.510670 22695 tablet_bootstrap.cc:654] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.510821 22696 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: No bootstrap required, opened a new log
I20260808 06:00:22.510926 22696 ts_tablet_manager.cc:1397] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:22.511268 22697 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: No bootstrap required, opened a new log
I20260808 06:00:22.511363 22697 ts_tablet_manager.cc:1397] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.511343 22696 raft_consensus.cc:348] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.511490 22696 raft_consensus.cc:374] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.511539 22696 raft_consensus.cc:729] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.511668 22696 consensus_queue.cc:260] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.511934 22697 raft_consensus.cc:348] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.512038 22697 raft_consensus.cc:374] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.512059 22696 ts_tablet_manager.cc:1428] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.512074 22697 raft_consensus.cc:729] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abff3f82b3946feabc50370a0c01450, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.512248 22695 tablet_bootstrap.cc:492] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: No bootstrap required, opened a new log
I20260808 06:00:22.512214 22697 consensus_queue.cc:260] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.512329 22695 ts_tablet_manager.cc:1397] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 06:00:22.512496 22697 ts_tablet_manager.cc:1428] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.512787 22695 raft_consensus.cc:348] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.512902 22695 raft_consensus.cc:374] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.512941 22695 raft_consensus.cc:729] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.513072 22695 consensus_queue.cc:260] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.513389 22695 ts_tablet_manager.cc:1428] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.566416 22702 raft_consensus.cc:482] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:22.566546 22702 raft_consensus.cc:504] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.567695 22702 leader_election.cc:288] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407)
I20260808 06:00:22.572835 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.568533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:52819 (local address 127.13.229.67:41407)
0808 06:00:22.568659 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.568663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.568732 (+ 69us) server_negotiation.cc:407] Connection header received
0808 06:00:22.568847 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.568849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.568917 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.568999 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:22.569191 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.569971 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.571462 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.572131 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.572266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.572329 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.572633 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.572705 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 06:00:22.572875 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.568080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:53373 (local address 127.13.229.65:38477)
0808 06:00:22.568335 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.568357 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.568375 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:00:22.568481 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.568484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.568535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.568627 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:22.568920 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.569786 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.571465 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.572165 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.572271 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.572326 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.572633 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.572709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
I20260808 06:00:22.573220 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.568419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41407 (local address 127.13.229.66:52819)
0808 06:00:22.568662 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:22.568674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:22.568754 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:22.569014 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:22.569016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:22.569031 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:22.569093 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.569096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.570068 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.570071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.571324 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.571336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.572288 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.572291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.572486 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.572520 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 06:00:22.572603 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:00:22.572724 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":123,"thread_start_us":71,"threads_started":1}
I20260808 06:00:22.573235 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.568005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:38477 (local address 127.13.229.66:53373)
0808 06:00:22.568270 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:22.568288 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:22.568382 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:22.568650 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:22.568653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:22.568673 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:22.568808 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.568815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.569912 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.569914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.571338 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:22.571344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.572287 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.572291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:22.572511 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.572527 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 06:00:22.572603 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 06:00:22.572724 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":120,"thread_start_us":109,"threads_started":1}
I20260808 06:00:22.573887 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57132eac2a704ba2900187fa79c15d35" candidate_uuid: "e92ffe25ce5d476888c2507ea4ef3571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450" is_pre_election: true
I20260808 06:00:22.573907 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57132eac2a704ba2900187fa79c15d35" candidate_uuid: "e92ffe25ce5d476888c2507ea4ef3571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c767ed77cd5744969eb50284181dd95d" is_pre_election: true
I20260808 06:00:22.574064 22327 raft_consensus.cc:2454] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92ffe25ce5d476888c2507ea4ef3571 in term 0.
I20260808 06:00:22.574136 22619 raft_consensus.cc:2454] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92ffe25ce5d476888c2507ea4ef3571 in term 0.
I20260808 06:00:22.574473 22404 leader_election.cc:302] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8abff3f82b3946feabc50370a0c01450, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:22.574774 22702 raft_consensus.cc:2781] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:22.574828 22702 raft_consensus.cc:482] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:22.574855 22702 raft_consensus.cc:3037] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.575834 22702 raft_consensus.cc:504] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.576232 22702 leader_election.cc:288] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [CANDIDATE]: Term 1 election: Requested vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407)
I20260808 06:00:22.576673 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57132eac2a704ba2900187fa79c15d35" candidate_uuid: "e92ffe25ce5d476888c2507ea4ef3571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450"
I20260808 06:00:22.576814 22327 raft_consensus.cc:3037] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.576797 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57132eac2a704ba2900187fa79c15d35" candidate_uuid: "e92ffe25ce5d476888c2507ea4ef3571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c767ed77cd5744969eb50284181dd95d"
I20260808 06:00:22.576908 22619 raft_consensus.cc:3037] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.577960 22327 raft_consensus.cc:2454] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92ffe25ce5d476888c2507ea4ef3571 in term 1.
I20260808 06:00:22.578095 22619 raft_consensus.cc:2454] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92ffe25ce5d476888c2507ea4ef3571 in term 1.
I20260808 06:00:22.578393 22404 leader_election.cc:302] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c767ed77cd5744969eb50284181dd95d, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:22.578624 22702 raft_consensus.cc:2781] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:22.578837 22702 raft_consensus.cc:686] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 LEADER]: Becoming Leader. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Running, Role: LEADER
I20260808 06:00:22.579103 22702 consensus_queue.cc:237] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.581324 22162 catalog_manager.cc:5495] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66). New cstate: current_term: 1 leader_uuid: "e92ffe25ce5d476888c2507ea4ef3571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } health_report { overall_health: UNKNOWN } } }
I20260808 06:00:22.593827 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.590096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.13.229.66:33951)
0808 06:00:22.590381 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.590390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.590409 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:00:22.590469 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.590471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.590518 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.590643 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:22.590896 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.591398 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.592337 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.592580 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.592695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.592974 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.593403 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.593540 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.593601 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.593650 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
I20260808 06:00:22.596220 22327 raft_consensus.cc:1261] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Refusing update from remote peer e92ffe25ce5d476888c2507ea4ef3571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.596531 22619 raft_consensus.cc:1261] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Refusing update from remote peer e92ffe25ce5d476888c2507ea4ef3571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.596709 22702 consensus_queue.cc:1035] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.597060 22768 consensus_queue.cc:1035] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.608115 22161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.608263 22161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.610117 22161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Sending DeleteTablet for 3 replicas of tablet 57132eac2a704ba2900187fa79c15d35
I20260808 06:00:22.610647 22305 tablet_service.cc:1514] Processing DeleteTablet for tablet 57132eac2a704ba2900187fa79c15d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:55782
I20260808 06:00:22.610838 22720 tablet_replica.cc:330] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: stopping tablet replica
I20260808 06:00:22.610888 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet 57132eac2a704ba2900187fa79c15d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:34224
I20260808 06:00:22.610916 22454 tablet_service.cc:1514] Processing DeleteTablet for tablet 57132eac2a704ba2900187fa79c15d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:45266
I20260808 06:00:22.610958 22720 raft_consensus.cc:2229] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.611053 22719 tablet_replica.cc:330] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: stopping tablet replica
I20260808 06:00:22.611068 22721 tablet_replica.cc:330] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: stopping tablet replica
I20260808 06:00:22.611088 22720 raft_consensus.cc:2258] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.611171 22719 raft_consensus.cc:2229] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.611186 22721 raft_consensus.cc:2229] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:22.611282 22719 raft_consensus.cc:2258] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.611446 22721 raft_consensus.cc:2258] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.612061 22161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 06:00:22.612197 22719 ts_tablet_manager.cc:1905] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.612197 22721 ts_tablet_manager.cc:1905] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 06:00:22.612447 22161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:22.614599 22719 ts_tablet_manager.cc:1918] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.614599 22721 ts_tablet_manager.cc:1918] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.614677 22721 log.cc:1196] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/57132eac2a704ba2900187fa79c15d35
I20260808 06:00:22.614677 22719 log.cc:1196] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/57132eac2a704ba2900187fa79c15d35
I20260808 06:00:22.614949 22721 ts_tablet_manager.cc:1939] T 57132eac2a704ba2900187fa79c15d35 P e92ffe25ce5d476888c2507ea4ef3571: Deleting consensus metadata
I20260808 06:00:22.614949 22719 ts_tablet_manager.cc:1939] T 57132eac2a704ba2900187fa79c15d35 P c767ed77cd5744969eb50284181dd95d: Deleting consensus metadata
I20260808 06:00:22.615654 22146 catalog_manager.cc:4841] TS e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951): tablet 57132eac2a704ba2900187fa79c15d35 (table test-table [id=384d5186da584b5187ec1db00d11e079]) successfully deleted
I20260808 06:00:22.616088 22148 catalog_manager.cc:4841] TS c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407): tablet 57132eac2a704ba2900187fa79c15d35 (table test-table [id=384d5186da584b5187ec1db00d11e079]) successfully deleted
I20260808 06:00:22.617002 22720 ts_tablet_manager.cc:1905] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.619485 22720 ts_tablet_manager.cc:1918] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:22.619563 22720 log.cc:1196] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/57132eac2a704ba2900187fa79c15d35
I20260808 06:00:22.619558 22597 tablet_service.cc:1467] Processing CreateTablet for tablet 0f638ab32d2c47428269bb16d040f053 (DEFAULT_TABLE table=test-table [id=08ef4e9516d64c69b169a34a6527cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.619629 22305 tablet_service.cc:1467] Processing CreateTablet for tablet 0f638ab32d2c47428269bb16d040f053 (DEFAULT_TABLE table=test-table [id=08ef4e9516d64c69b169a34a6527cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.619807 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f638ab32d2c47428269bb16d040f053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.619824 22720 ts_tablet_manager.cc:1939] T 57132eac2a704ba2900187fa79c15d35 P 8abff3f82b3946feabc50370a0c01450: Deleting consensus metadata
I20260808 06:00:22.619942 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f638ab32d2c47428269bb16d040f053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.621797 22697 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Bootstrap starting.
I20260808 06:00:22.622009 22148 catalog_manager.cc:4841] TS 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477): tablet 57132eac2a704ba2900187fa79c15d35 (table test-table [id=384d5186da584b5187ec1db00d11e079]) successfully deleted
I20260808 06:00:22.622124 22696 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Bootstrap starting.
I20260808 06:00:22.622761 22697 tablet_bootstrap.cc:654] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.622912 22696 tablet_bootstrap.cc:654] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.623945 22697 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: No bootstrap required, opened a new log
I20260808 06:00:22.624034 22697 ts_tablet_manager.cc:1397] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:22.624583 22696 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: No bootstrap required, opened a new log
I20260808 06:00:22.624519 22697 raft_consensus.cc:348] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.624626 22697 raft_consensus.cc:374] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.624658 22697 raft_consensus.cc:729] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abff3f82b3946feabc50370a0c01450, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.624665 22696 ts_tablet_manager.cc:1397] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.624792 22697 consensus_queue.cc:260] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.625051 22697 ts_tablet_manager.cc:1428] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.625061 22696 raft_consensus.cc:348] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.625157 22696 raft_consensus.cc:374] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.625190 22696 raft_consensus.cc:729] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.625319 22696 consensus_queue.cc:260] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.625411 22454 tablet_service.cc:1467] Processing CreateTablet for tablet 0f638ab32d2c47428269bb16d040f053 (DEFAULT_TABLE table=test-table [id=08ef4e9516d64c69b169a34a6527cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:22.625595 22696 ts_tablet_manager.cc:1428] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.625655 22454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f638ab32d2c47428269bb16d040f053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:22.627625 22695 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Bootstrap starting.
I20260808 06:00:22.628556 22695 tablet_bootstrap.cc:654] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:22.630250 22695 tablet_bootstrap.cc:492] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: No bootstrap required, opened a new log
I20260808 06:00:22.630342 22695 ts_tablet_manager.cc:1397] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:22.630743 22695 raft_consensus.cc:348] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.630848 22695 raft_consensus.cc:374] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:22.630899 22695 raft_consensus.cc:729] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92ffe25ce5d476888c2507ea4ef3571, State: Initialized, Role: FOLLOWER
I20260808 06:00:22.631012 22695 consensus_queue.cc:260] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.631266 22695 ts_tablet_manager.cc:1428] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:00:22.719782 22703 raft_consensus.cc:482] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:22.719982 22703 raft_consensus.cc:504] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.720515 22703 leader_election.cc:288] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.720957 22472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f638ab32d2c47428269bb16d040f053" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571" is_pre_election: true
I20260808 06:00:22.720986 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f638ab32d2c47428269bb16d040f053" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450" is_pre_election: true
I20260808 06:00:22.721125 22472 raft_consensus.cc:2454] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.721143 22327 raft_consensus.cc:2454] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 0.
I20260808 06:00:22.721462 22549 leader_election.cc:302] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8abff3f82b3946feabc50370a0c01450, c767ed77cd5744969eb50284181dd95d; no voters:
I20260808 06:00:22.721732 22703 raft_consensus.cc:2781] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:22.721791 22703 raft_consensus.cc:482] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:22.721828 22703 raft_consensus.cc:3037] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.723081 22703 raft_consensus.cc:504] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.723457 22703 leader_election.cc:288] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Requested vote from peers 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477), e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951)
I20260808 06:00:22.723851 22327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f638ab32d2c47428269bb16d040f053" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8abff3f82b3946feabc50370a0c01450"
I20260808 06:00:22.723881 22472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f638ab32d2c47428269bb16d040f053" candidate_uuid: "c767ed77cd5744969eb50284181dd95d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92ffe25ce5d476888c2507ea4ef3571"
I20260808 06:00:22.724020 22327 raft_consensus.cc:3037] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.724041 22472 raft_consensus.cc:3037] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:22.724952 22472 raft_consensus.cc:2454] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.725242 22327 raft_consensus.cc:2454] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c767ed77cd5744969eb50284181dd95d in term 1.
I20260808 06:00:22.725238 22547 leader_election.cc:302] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c767ed77cd5744969eb50284181dd95d, e92ffe25ce5d476888c2507ea4ef3571; no voters:
I20260808 06:00:22.725437 22703 raft_consensus.cc:2781] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:22.725525 22703 raft_consensus.cc:686] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Becoming Leader. State: Replica: c767ed77cd5744969eb50284181dd95d, State: Running, Role: LEADER
I20260808 06:00:22.725871 22703 consensus_queue.cc:237] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } }
I20260808 06:00:22.727489 22162 catalog_manager.cc:5495] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d reported cstate change: term changed from 0 to 1, leader changed from <none> to c767ed77cd5744969eb50284181dd95d (127.13.229.67). New cstate: current_term: 1 leader_uuid: "c767ed77cd5744969eb50284181dd95d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c767ed77cd5744969eb50284181dd95d" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41407 } health_report { overall_health: HEALTHY } } }
I20260808 06:00:22.758997 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:22.755725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.13.229.67:41407)
0808 06:00:22.755844 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:22.755848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:22.756076 (+ 228us) server_negotiation.cc:407] Connection header received
0808 06:00:22.756116 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:22.756118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:22.756157 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:22.756241 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:22.756484 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.757125 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.757958 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:22.758186 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:22.758305 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:22.758554 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:22.758629 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:22.758710 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:22.758821 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 06:00:22.758869 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:00:22.765703 22327 raft_consensus.cc:1261] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.766140 22716 consensus_queue.cc:1035] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8abff3f82b3946feabc50370a0c01450" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.766530 22472 raft_consensus.cc:1261] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Refusing update from remote peer c767ed77cd5744969eb50284181dd95d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:22.766880 22703 consensus_queue.cc:1035] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92ffe25ce5d476888c2507ea4ef3571" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:22.796176 22162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.796322 22162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:22.798147 22162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ebca1d49748cc97568a0a27be882b: Sending DeleteTablet for 3 replicas of tablet 0f638ab32d2c47428269bb16d040f053
I20260808 06:00:22.798669 22305 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f638ab32d2c47428269bb16d040f053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:55782
I20260808 06:00:22.798782 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f638ab32d2c47428269bb16d040f053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:34224
I20260808 06:00:22.798820 22454 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f638ab32d2c47428269bb16d040f053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:22 UTC) from {username='oryx', principal='oryx/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:45266
I20260808 06:00:22.798877 22720 tablet_replica.cc:330] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: stopping tablet replica
I20260808 06:00:22.798969 22719 tablet_replica.cc:330] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: stopping tablet replica
I20260808 06:00:22.798969 22721 tablet_replica.cc:330] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: stopping tablet replica
I20260808 06:00:22.798985 22720 raft_consensus.cc:2229] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.799077 22719 raft_consensus.cc:2229] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:22.799077 22721 raft_consensus.cc:2229] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:22.799124 22720 raft_consensus.cc:2258] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.799181 22721 raft_consensus.cc:2258] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.799261 22719 raft_consensus.cc:2258] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:22.799852 22721 ts_tablet_manager.cc:1905] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.799872 22719 ts_tablet_manager.cc:1905] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.800040 22720 ts_tablet_manager.cc:1905] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:22.802354 22719 ts_tablet_manager.cc:1918] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.802354 22721 ts_tablet_manager.cc:1918] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.802385 22720 ts_tablet_manager.cc:1918] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 06:00:22.802440 22721 log.cc:1196] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/0f638ab32d2c47428269bb16d040f053
I20260808 06:00:22.802440 22719 log.cc:1196] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/0f638ab32d2c47428269bb16d040f053
I20260808 06:00:22.802479 22720 log.cc:1196] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/0f638ab32d2c47428269bb16d040f053
I20260808 06:00:22.802711 22721 ts_tablet_manager.cc:1939] T 0f638ab32d2c47428269bb16d040f053 P e92ffe25ce5d476888c2507ea4ef3571: Deleting consensus metadata
I20260808 06:00:22.802711 22719 ts_tablet_manager.cc:1939] T 0f638ab32d2c47428269bb16d040f053 P c767ed77cd5744969eb50284181dd95d: Deleting consensus metadata
I20260808 06:00:22.802788 22720 ts_tablet_manager.cc:1939] T 0f638ab32d2c47428269bb16d040f053 P 8abff3f82b3946feabc50370a0c01450: Deleting consensus metadata
I20260808 06:00:22.803422 22148 catalog_manager.cc:4841] TS 8abff3f82b3946feabc50370a0c01450 (127.13.229.65:38477): tablet 0f638ab32d2c47428269bb16d040f053 (table test-table [id=08ef4e9516d64c69b169a34a6527cc1f]) successfully deleted
I20260808 06:00:22.803591 22148 catalog_manager.cc:4841] TS c767ed77cd5744969eb50284181dd95d (127.13.229.67:41407): tablet 0f638ab32d2c47428269bb16d040f053 (table test-table [id=08ef4e9516d64c69b169a34a6527cc1f]) successfully deleted
I20260808 06:00:22.803805 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22236
I20260808 06:00:22.803844 22146 catalog_manager.cc:4841] TS e92ffe25ce5d476888c2507ea4ef3571 (127.13.229.66:33951): tablet 0f638ab32d2c47428269bb16d040f053 (table test-table [id=08ef4e9516d64c69b169a34a6527cc1f]) successfully deleted
I20260808 06:00:22.813354 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22385
I20260808 06:00:22.821755 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22531
I20260808 06:00:22.830139 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22130
2026-08-08T06:00:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3235 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22786](info): set up 2 sockets
Aug 08 06:00:22 dist-test-slave-zkpd krb5kdc[22786](info): commencing operation
krb5kdc: starting...
W20260808 06:00:23.897554 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:23 dist-test-slave-zkpd krb5kdc[22786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:24.023322 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34747
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:24.135339 22800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:24.135598 22800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:24.135641 22800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:24.139647 22800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:24.139698 22800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:24.139717 22800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:24.139734 22800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:24.139783 22800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:24.144976 22800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:24.146030 22800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:24.147048 22800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:24.155619 22806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:24.155628 22808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:24.155628 22805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:24.155966 22800 server_base.cc:1034] running on GCE node
I20260808 06:00:24.156563 22800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:24.157651 22800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:24.158841 22800 hybrid_clock.cc:648] HybridClock initialized: now 1786168824158829 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T06:00:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1339 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:24 dist-test-slave-zkpd krb5kdc[22813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:24 dist-test-slave-zkpd krb5kdc[22813](info): set up 2 sockets
Aug 08 06:00:24 dist-test-slave-zkpd krb5kdc[22813](info): commencing operation
krb5kdc: starting...
W20260808 06:00:25.290553 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:25 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:25.414340 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34497
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:25.526376 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:25.526650 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:25.526732 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:25.530644 22826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:25.530722 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:25.530771 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:25.530845 22826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:25.530881 22826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:25.535982 22826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:25.536949 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:25.537901 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:25.546192 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:25.546185 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:25.546166 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:25.546525 22826 server_base.cc:1034] running on GCE node
I20260808 06:00:25.547133 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:25.548257 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:25.549456 22826 hybrid_clock.cc:648] HybridClock initialized: now 1786168825549436 us; error 30 us; skew 500 ppm
Aug 08 06:00:25 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168825, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:25.552150 22826 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:25.553005 22826 webserver.cc:460] Webserver started at http://127.13.229.126:34825/ using document root <none> and password file <none>
I20260808 06:00:25.553560 22826 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:25.553651 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:25.553906 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:25.555886 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "886a6d7d980749c29c28af0fd3478934"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "a0f757553c981d0393234e18255ee506"
server_key_iv: "06e36fec3b78ac66b54b6103e4b1a430"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.556507 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "886a6d7d980749c29c28af0fd3478934"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "a0f757553c981d0393234e18255ee506"
server_key_iv: "06e36fec3b78ac66b54b6103e4b1a430"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.560225 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:25.562501 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:25.563668 22826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:25.563833 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "886a6d7d980749c29c28af0fd3478934"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "a0f757553c981d0393234e18255ee506"
server_key_iv: "06e36fec3b78ac66b54b6103e4b1a430"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.564002 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:25.574869 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:25.577590 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:25.577791 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:25.585039 22826 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:34497
I20260808 06:00:25.585067 22898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:34497 every 8 connection(s)
I20260808 06:00:25.585971 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:25.588680 22899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:25.590049 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22826
I20260808 06:00:25.590196 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:25.590513 14229 external_mini_cluster.cc:1428] Setting key 8add7d7f16b23729b90964320f74cf2c
I20260808 06:00:25.595353 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Bootstrap starting.
I20260808 06:00:25.597873 22899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:25.598722 22899 log.cc:824] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:00:25 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:25.600984 22899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: No bootstrap required, opened a new log
I20260808 06:00:25.604923 22899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:25.605170 22899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:25.605258 22899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a6d7d980749c29c28af0fd3478934, State: Initialized, Role: FOLLOWER
I20260808 06:00:25.605520 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:25.591884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.13.229.126:34497)
0808 06:00:25.592346 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:25.592365 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:25.592407 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:00:25.593252 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:25.593278 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:25.593689 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:25.594068 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:25.594715 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.596153 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.597039 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.597698 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.597895 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:25.600731 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:25.600744 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:25.600758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:25.600786 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:25.602512 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:25.603071 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:25.603075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:25.603080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:25.603186 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:25.603482 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:25.603484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:25.603485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:25.603876 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:25.604096 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:25.604367 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 06:00:25.604658 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260808 06:00:25.605741 22899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:25.605902 22899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:25.605988 22899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:25.606088 22899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:25.607290 22899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:25.607661 22899 leader_election.cc:302] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886a6d7d980749c29c28af0fd3478934; no voters:
I20260808 06:00:25.608013 22899 leader_election.cc:288] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:25.608079 22904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:25.608314 22904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 LEADER]: Becoming Leader. State: Replica: 886a6d7d980749c29c28af0fd3478934, State: Running, Role: LEADER
I20260808 06:00:25.608637 22904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:25.609217 22899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:25.610740 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886a6d7d980749c29c28af0fd3478934. Latest consensus state: current_term: 1 leader_uuid: "886a6d7d980749c29c28af0fd3478934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } } }
I20260808 06:00:25.610957 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:25.610997 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886a6d7d980749c29c28af0fd3478934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } } }
I20260808 06:00:25.611523 22915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:25.611665 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:25.614951 22915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:25.619940 22915 catalog_manager.cc:1347] Generated new cluster ID: 8b10a83fa0fa47f6bab62af434ca2f1d
I20260808 06:00:25.620002 22915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:25.632283 22915 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:25.633455 22915 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:25.641858 22915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Generated new TSK 0
I20260808 06:00:25.642577 22915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:25.675216 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:25.788142 22925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:25.788394 22925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:25.788436 22925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:25.792500 22925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:25.792549 22925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:25.792632 22925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:25.797732 22925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:25.798713 22925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:25.799741 22925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:25.809103 22932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:25.809103 22935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:25.809134 22931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:25.809335 22925 server_base.cc:1034] running on GCE node
I20260808 06:00:25.809916 22925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:25.810654 22925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:25.811843 22925 hybrid_clock.cc:648] HybridClock initialized: now 1786168825811831 us; error 22 us; skew 500 ppm
Aug 08 06:00:25 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168825, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:25.814509 22925 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:25.815315 22925 webserver.cc:460] Webserver started at http://127.13.229.65:36571/ using document root <none> and password file <none>
I20260808 06:00:25.815855 22925 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:25.815999 22925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:25.816229 22925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:25.818173 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "862d3102d3b144b9908bbdc270d5b191"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "d52315447a8eec65f700e814b691ffda"
server_key_iv: "292bc07924c762ad6b8640afbf8a0da2"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.818739 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "862d3102d3b144b9908bbdc270d5b191"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "d52315447a8eec65f700e814b691ffda"
server_key_iv: "292bc07924c762ad6b8640afbf8a0da2"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.822268 22925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:25.824585 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:25.825798 22925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:25.825946 22925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "862d3102d3b144b9908bbdc270d5b191"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "d52315447a8eec65f700e814b691ffda"
server_key_iv: "292bc07924c762ad6b8640afbf8a0da2"
server_key_version: "encryptionkey@0"
I20260808 06:00:25.826052 22925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:25.837182 22925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:25.839823 22925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:25.840063 22925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:25.840597 22925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:25.841389 22925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:25.841459 22925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:25.841531 22925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:25.841575 22925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:25.849843 22925 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:38647
I20260808 06:00:25.849922 23057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:38647 every 8 connection(s)
I20260808 06:00:25.850699 22925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:25.851431 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 22925
I20260808 06:00:25.851565 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:25.851917 14229 external_mini_cluster.cc:1428] Setting key ff093f6e50a4c64fdd2ac23e9cbbd5f0
Aug 08 06:00:25 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168825, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:25.866075 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:25.852736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:45943 (local address 127.13.229.126:34497)
0808 06:00:25.852847 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:25.852856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:25.853965 (+ 1109us) server_negotiation.cc:407] Connection header received
0808 06:00:25.855909 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:25.855914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:25.855977 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:25.856073 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:25.857171 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.858055 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.859335 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.859805 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.859980 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:25.862441 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:25.862447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:25.862452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:25.862489 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:25.864174 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:25.864636 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:25.864638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:25.864639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:25.864714 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:25.865022 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:25.865025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:25.865026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:25.865200 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:25.865268 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:25.865798 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 06:00:25.865940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:00:25.867023 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:25.853239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:45943)
0808 06:00:25.853790 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:25.853847 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:25.855628 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:25.856223 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:25.856229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:25.856571 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:25.857019 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:25.857032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.858232 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.858235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:25.859174 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:25.859184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:25.860017 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:25.860020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:25.860165 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:25.860793 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:25.860809 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:25.862253 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:25.864283 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:25.864288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:25.864300 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:25.864547 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:25.864800 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:25.864802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:25.864803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:25.864944 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:25.865322 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:25.865328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:25.865580 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 06:00:25.865734 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 06:00:25.866103 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
I20260808 06:00:25.868458 23058 heartbeater.cc:347] Connected to a master server at 127.13.229.126:34497
I20260808 06:00:25.868733 23058 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:25.869356 23058 heartbeater.cc:511] Master 127.13.229.126:34497 requested a full tablet report, sending...
I20260808 06:00:25.871043 22859 ts_manager.cc:194] Registered new tserver with Master: 862d3102d3b144b9908bbdc270d5b191 (127.13.229.65:38647)
I20260808 06:00:25.872268 22859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:25.884155 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:25.994936 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:25.995179 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:25.995256 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:25.999133 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:25.999185 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:25.999280 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:26.004247 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:26.005273 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:26.006299 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:26.016119 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.016119 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.016134 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.016455 23063 server_base.cc:1034] running on GCE node
I20260808 06:00:26.016930 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:26.017563 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:26.018728 23063 hybrid_clock.cc:648] HybridClock initialized: now 1786168826018691 us; error 49 us; skew 500 ppm
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:26.021421 23063 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:26.022214 23063 webserver.cc:460] Webserver started at http://127.13.229.66:40169/ using document root <none> and password file <none>
I20260808 06:00:26.022746 23063 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:26.022816 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:26.023054 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:26.025125 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "1fc31cbe46b74552a570f0d8a028e820"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "8dfd6a1f683995f370ab1a2b4f1d2a12"
server_key_iv: "e807481eab12c7a761c3ac1d3bf9f6e3"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.025707 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "1fc31cbe46b74552a570f0d8a028e820"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "8dfd6a1f683995f370ab1a2b4f1d2a12"
server_key_iv: "e807481eab12c7a761c3ac1d3bf9f6e3"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.029559 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 06:00:26.032050 23079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.033227 23063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:00:26.033352 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "1fc31cbe46b74552a570f0d8a028e820"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "8dfd6a1f683995f370ab1a2b4f1d2a12"
server_key_iv: "e807481eab12c7a761c3ac1d3bf9f6e3"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.033450 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:26.052868 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:26.055814 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:26.056036 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:26.056629 23063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:26.057464 23063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:26.057510 23063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.057579 23063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:26.057619 23063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.066720 23063 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:38395
I20260808 06:00:26.066744 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:38395 every 8 connection(s)
I20260808 06:00:26.067691 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:26.070796 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23063
I20260808 06:00:26.070950 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:26.071244 14229 external_mini_cluster.cc:1428] Setting key a7d740354213bfd95a81300165370038
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:00:26.082937 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.069562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:37661 (local address 127.13.229.126:34497)
0808 06:00:26.069701 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.069705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.070541 (+ 836us) server_negotiation.cc:407] Connection header received
0808 06:00:26.072006 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.072009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.072067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.072158 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.073495 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.074348 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.076089 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.076303 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.076381 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.079217 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:26.079220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:26.079222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:26.079246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:26.080889 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:26.081412 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:26.081416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:26.081418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:26.081500 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:26.081843 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:26.081847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:26.081849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:26.082030 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:26.082139 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:26.082641 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 06:00:26.082761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 06:00:26.083740 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.069941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:37661)
0808 06:00:26.070378 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.070419 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.071720 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.072341 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.072347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.072858 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.073328 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.073342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.074482 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.074484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.075935 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.075952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.076458 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.076464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.076700 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.077340 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.077366 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:26.079041 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:26.081013 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:26.081019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:26.081033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:26.081296 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:26.081596 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:26.081600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:26.081602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:26.081745 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:26.082138 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:26.082144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:26.082393 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 06:00:26.082584 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 06:00:26.082943 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:00:26.085033 23198 heartbeater.cc:347] Connected to a master server at 127.13.229.126:34497
I20260808 06:00:26.085279 23198 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:26.085757 23198 heartbeater.cc:511] Master 127.13.229.126:34497 requested a full tablet report, sending...
I20260808 06:00:26.086930 22859 ts_manager.cc:194] Registered new tserver with Master: 1fc31cbe46b74552a570f0d8a028e820 (127.13.229.66:38395)
I20260808 06:00:26.087752 22859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:37661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:26.100628 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:26.210561 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:26.210783 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:26.210824 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:26.214793 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:26.214843 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:26.214900 23203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:26.219856 23203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:26.220809 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:26.221730 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:26.230978 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.231047 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.230983 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.231488 23203 server_base.cc:1034] running on GCE node
I20260808 06:00:26.231935 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:26.232492 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:26.233660 23203 hybrid_clock.cc:648] HybridClock initialized: now 1786168826233640 us; error 31 us; skew 500 ppm
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:26.236512 23203 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:26.237445 23203 webserver.cc:460] Webserver started at http://127.13.229.67:41887/ using document root <none> and password file <none>
I20260808 06:00:26.237941 23203 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:26.237998 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:26.238150 23203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:26.240134 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "a13761cd472b4273bd07ee2458664290"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "ed4353fb246e35443913b222bd23c774"
server_key_iv: "f73ece22ba6173d0a8cf7a0df5bdad6a"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.240655 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "a13761cd472b4273bd07ee2458664290"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "ed4353fb246e35443913b222bd23c774"
server_key_iv: "f73ece22ba6173d0a8cf7a0df5bdad6a"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.244161 23203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:26.246431 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.247642 23203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:26.247736 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "a13761cd472b4273bd07ee2458664290"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "ed4353fb246e35443913b222bd23c774"
server_key_iv: "f73ece22ba6173d0a8cf7a0df5bdad6a"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.247820 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:26.261907 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:26.265117 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:26.265265 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:26.265780 23203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:26.266538 23203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:26.266583 23203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.266620 23203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:26.266640 23203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.275815 23203 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:43291
I20260808 06:00:26.275861 23339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:43291 every 8 connection(s)
I20260808 06:00:26.276705 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:26.277359 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23203
I20260808 06:00:26.277520 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:26.277804 14229 external_mini_cluster.cc:1428] Setting key c76979d10e441f6e133998089709ed5e
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:26.289857 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.278543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54809 (local address 127.13.229.126:34497)
0808 06:00:26.278673 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.278677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.279636 (+ 959us) server_negotiation.cc:407] Connection header received
0808 06:00:26.281043 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.281045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.281091 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.281158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.282211 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.282713 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.284100 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.284319 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.284409 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.286828 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:26.286831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:26.286833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:26.286857 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:26.288103 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:26.288577 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:26.288579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:26.288580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:26.288652 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:26.288970 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:26.288972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:26.288973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:26.289138 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:26.289213 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:26.289641 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 06:00:26.289737 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:00:26.290861 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.278956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:54809)
0808 06:00:26.279486 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.279543 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.280847 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.281294 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.281300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.281632 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.282089 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.282102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.282984 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.282987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.283967 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.283979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.284454 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.284457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.284600 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.285248 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.285265 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:26.286679 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:26.288218 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:26.288223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:26.288235 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:26.288492 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:26.288731 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:26.288733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:26.288735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:26.288889 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:26.289228 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:26.289233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:26.289444 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 06:00:26.289596 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 06:00:26.289861 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260808 06:00:26.292057 23340 heartbeater.cc:347] Connected to a master server at 127.13.229.126:34497
I20260808 06:00:26.292309 23340 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:26.292790 23340 heartbeater.cc:511] Master 127.13.229.126:34497 requested a full tablet report, sending...
I20260808 06:00:26.293797 22859 ts_manager.cc:194] Registered new tserver with Master: a13761cd472b4273bd07ee2458664290 (127.13.229.67:43291)
I20260808 06:00:26.294368 22859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:54809
I20260808 06:00:26.303786 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
Entry for principal oryx/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab.
WARNING: no policy specified for oryx/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
Entry for principal oryx/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab.
WARNING: no policy specified for oryx/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
Entry for principal oryx/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab.
I20260808 06:00:26.378825 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22925
I20260808 06:00:26.386257 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23063
I20260808 06:00:26.392933 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23203
I20260808 06:00:26.399474 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 22826
I20260808 06:00:26.407859 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34497
--webserver_interface=127.13.229.126
--webserver_port=34825
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:26.517788 23349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:26.518064 23349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:26.518132 23349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:26.522037 23349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:26.522114 23349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:26.522151 23349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:26.522207 23349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:26.522244 23349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:26.527292 23349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=34825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:26.528373 23349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:26.529340 23349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:26.537539 23355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.537531 23357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.537532 23354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.537748 23349 server_base.cc:1034] running on GCE node
I20260808 06:00:26.538281 23349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:26.539414 23349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:26.540606 23349 hybrid_clock.cc:648] HybridClock initialized: now 1786168826540596 us; error 28 us; skew 500 ppm
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:26.543442 23349 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:26.544363 23349 webserver.cc:460] Webserver started at http://127.13.229.126:34825/ using document root <none> and password file <none>
I20260808 06:00:26.544912 23349 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:26.545019 23349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:26.548795 23349 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 06:00:26.550809 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.552053 23349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:26.552188 23349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "886a6d7d980749c29c28af0fd3478934"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "a0f757553c981d0393234e18255ee506"
server_key_iv: "06e36fec3b78ac66b54b6103e4b1a430"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.552722 23349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:26.566313 23349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:26.569615 23349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:26.569821 23349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:26.577136 23349 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:34497
I20260808 06:00:26.577165 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:34497 every 8 connection(s)
I20260808 06:00:26.578177 23349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:26.582836 23419 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:00:26.584250 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23349
I20260808 06:00:26.584543 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Bootstrap starting.
I20260808 06:00:26.584738 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:38647
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:34497
--webserver_port=36571
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:00:26.607483 23419 log.cc:824] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:26.616070 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:00:26.616473 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Bootstrap complete.
I20260808 06:00:26.620844 23419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:26.621234 23419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886a6d7d980749c29c28af0fd3478934, State: Initialized, Role: FOLLOWER
I20260808 06:00:26.621779 23419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:26.621922 23419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:26.621987 23419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:26.622103 23419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:00:26.624069 23419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:26.624409 23419 leader_election.cc:302] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886a6d7d980749c29c28af0fd3478934; no voters:
I20260808 06:00:26.624768 23419 leader_election.cc:288] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:00:26.624855 23423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:00:26.625039 23423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [term 2 LEADER]: Becoming Leader. State: Replica: 886a6d7d980749c29c28af0fd3478934, State: Running, Role: LEADER
I20260808 06:00:26.625327 23423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } }
I20260808 06:00:26.626003 23419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:26.626269 23423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886a6d7d980749c29c28af0fd3478934. Latest consensus state: current_term: 2 leader_uuid: "886a6d7d980749c29c28af0fd3478934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } } }
I20260808 06:00:26.626387 23423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:26.626734 23428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:26.626778 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "886a6d7d980749c29c28af0fd3478934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886a6d7d980749c29c28af0fd3478934" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34497 } } }
I20260808 06:00:26.627002 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934 [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:26.630391 23428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:26.631055 23428 catalog_manager.cc:1259] Loaded cluster ID: 8b10a83fa0fa47f6bab62af434ca2f1d
I20260808 06:00:26.631134 23428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:26.632840 23428 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:26.633387 23428 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 886a6d7d980749c29c28af0fd3478934: Loaded TSK: 0
I20260808 06:00:26.634322 23428 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:00:26.719997 23421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:26.720263 23421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:26.720335 23421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:26.724387 23421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:26.724458 23421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:26.724543 23421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:26.730154 23421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.65.keytab
--principal=oryx/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:38647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=36571
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:26.731135 23421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:26.732221 23421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:26.741457 23447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.741483 23449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.741503 23446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.741873 23421 server_base.cc:1034] running on GCE node
I20260808 06:00:26.742290 23421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:26.742964 23421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:26.744134 23421 hybrid_clock.cc:648] HybridClock initialized: now 1786168826744112 us; error 35 us; skew 500 ppm
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:26.746894 23421 init.cc:377] Logged in from keytab as oryx/127.13.229.65@KRBTEST.COM (short username oryx)
I20260808 06:00:26.747820 23421 webserver.cc:460] Webserver started at http://127.13.229.65:36571/ using document root <none> and password file <none>
I20260808 06:00:26.748383 23421 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:26.748458 23421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:26.752373 23421 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:26.754506 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.755847 23421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:00:26.756008 23421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "862d3102d3b144b9908bbdc270d5b191"
format_stamp: "Formatted at 2026-08-08 06:00:25 on dist-test-slave-zkpd"
server_key: "d52315447a8eec65f700e814b691ffda"
server_key_iv: "292bc07924c762ad6b8640afbf8a0da2"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.756611 23421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:26.778589 23421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:26.781736 23421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:26.781951 23421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:26.782558 23421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:26.783435 23421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:26.783507 23421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.783586 23421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:26.783640 23421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.792268 23421 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:38647
I20260808 06:00:26.792310 23575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:38647 every 8 connection(s)
I20260808 06:00:26.793263 23421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:26.803102 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23421
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:26.803548 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:38395
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:34497
--webserver_port=40169
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:26.805540 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.795435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:49811)
0808 06:00:26.795877 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.795949 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.797123 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.798146 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.798152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.798489 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.798928 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.798942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.800180 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.800183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.801020 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.801030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.801706 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.801709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.801840 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.802343 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.802358 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:26.804730 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":130,"threads_started":1}
W20260808 06:00:26.805907 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:26.795394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:49811 (local address 127.13.229.126:34497)
0808 06:00:26.795867 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.795885 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.796065 (+ 180us) server_negotiation.cc:407] Connection header received
0808 06:00:26.797489 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.797500 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.797792 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.798058 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.799087 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.800054 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.801128 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.801576 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.801710 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.805023 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:49811: BlockingRecv error: recv got EOF from 127.13.229.65:49811 (error 108)
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
W20260808 06:00:26.806433 23576 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:34497 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.229.126:34497: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:26.811951 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.806889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:44279)
0808 06:00:26.807004 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.807020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.807098 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.807347 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.807350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.807504 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.807586 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.807592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.808359 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.808364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.808896 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.808910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.809363 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.809367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.809458 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.809945 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.809956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:26.811797 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 06:00:26.812040 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:26.806985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:44279 (local address 127.13.229.126:34497)
0808 06:00:26.807124 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.807131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.807146 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:00:26.807181 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.807184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.807230 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.807332 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.807698 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.808242 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.809024 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.809239 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.809350 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.811928 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:44279: BlockingRecv error: recv got EOF from 127.13.229.65:44279 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:26.817163 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.812461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:48285)
0808 06:00:26.812550 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.812564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.812655 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.812873 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.812876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.813023 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.813086 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.813091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.813784 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.813789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.814283 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.814290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.814726 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.814730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.814820 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.815259 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.815271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:26.817033 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 06:00:26.817250 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:26.812554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48285 (local address 127.13.229.126:34497)
0808 06:00:26.812672 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.812678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.812693 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:00:26.812729 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.812731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.812775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.812857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.813193 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.813657 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.814401 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.814608 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.814724 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.817145 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:48285: BlockingRecv error: recv got EOF from 127.13.229.65:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 06:00:26.817353 23576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:00:26.915879 23580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:26.916149 23580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:26.916191 23580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:26.920249 23580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:26.920321 23580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:26.920413 23580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:26.925613 23580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.66.keytab
--principal=oryx/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:38395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=40169
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:26.926585 23580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:26.927695 23580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:26.937412 23586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:26.937426 23585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.937465 23580 server_base.cc:1034] running on GCE node
W20260808 06:00:26.937422 23588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:26.938360 23580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:26.938984 23580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:26.940197 23580 hybrid_clock.cc:648] HybridClock initialized: now 1786168826940182 us; error 24 us; skew 500 ppm
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168826, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:26.942915 23580 init.cc:377] Logged in from keytab as oryx/127.13.229.66@KRBTEST.COM (short username oryx)
I20260808 06:00:26.943747 23580 webserver.cc:460] Webserver started at http://127.13.229.66:40169/ using document root <none> and password file <none>
I20260808 06:00:26.944458 23580 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:26.944562 23580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:26.948405 23580 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 06:00:26.950465 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.951668 23580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:26.951766 23580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "1fc31cbe46b74552a570f0d8a028e820"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "8dfd6a1f683995f370ab1a2b4f1d2a12"
server_key_iv: "e807481eab12c7a761c3ac1d3bf9f6e3"
server_key_version: "encryptionkey@0"
I20260808 06:00:26.952364 23580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:26.975113 23580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:26.978508 23580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:26.978660 23580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:26.979230 23580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:26.980049 23580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:26.980100 23580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.980140 23580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:26.980165 23580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:26.988873 23580 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:38395
I20260808 06:00:26.988909 23716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:38395 every 8 connection(s)
I20260808 06:00:26.989902 23580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:26.991286 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23580
I20260808 06:00:26.991818 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:43291
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:34497
--webserver_port=41887
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:26 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:00:27.002607 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:26.991782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:50263 (local address 127.13.229.126:34497)
0808 06:00:26.991971 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:26.991978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:26.993032 (+ 1054us) server_negotiation.cc:407] Connection header received
0808 06:00:26.994492 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:26.994496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:26.994571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:26.994669 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:26.995831 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.996425 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.998436 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.998647 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.998725 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.002489 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:50263: BlockingRecv error: recv got EOF from 127.13.229.66:50263 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:00:27.003203 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:26.992364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:50263)
0808 06:00:26.992869 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:26.992911 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:26.994264 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:26.994848 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:26.994855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:26.995203 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:26.995678 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.995690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.996605 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.996612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.998296 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:26.998312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:26.998818 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:26.998825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:26.999071 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:26.999691 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:26.999716 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.002367 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":149,"threads_started":1}
W20260808 06:00:27.003767 23717 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:34497 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.229.126:34497: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:27.009387 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.004268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:58627)
0808 06:00:27.004393 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.004409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.004482 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.004692 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.004694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.004817 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.004903 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.004908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.005915 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.005917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.006444 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.006451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.007004 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.007006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.007072 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.007546 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.007557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.009295 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 06:00:27.009557 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.004369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:58627 (local address 127.13.229.126:34497)
0808 06:00:27.004488 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.004492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.004506 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:27.004547 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.004550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.004589 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.004671 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.005008 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.005798 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.006547 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.006897 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.006994 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.009437 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:58627: BlockingRecv error: recv got EOF from 127.13.229.66:58627 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:27.014636 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.009963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:40873)
0808 06:00:27.010060 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.010075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.010169 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.010419 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.010422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.010562 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.010623 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.010628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.011325 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.011330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.011856 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.011865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.012351 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.012355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.012449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.012899 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.012911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.014526 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 06:00:27.014842 23717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:00:27.014869 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.010070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:40873 (local address 127.13.229.126:34497)
0808 06:00:27.010204 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.010210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.010224 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:27.010256 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.010258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.010291 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.010402 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.010733 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.011208 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.012019 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.012243 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.012350 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.014631 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:40873: BlockingRecv error: recv got EOF from 127.13.229.66:40873 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 06:00:27.106065 23720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:27.106312 23720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:27.106352 23720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:27.110267 23720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:27.110316 23720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:27.110390 23720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:27.115527 23720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/krb5kdc/oryx_127.13.229.67.keytab
--principal=oryx/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:43291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=41887
--tserver_master_addrs=127.13.229.126:34497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:27.116730 23720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:27.117803 23720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:27.128192 23728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:27.128226 23726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:27.128226 23725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:27.128468 23720 server_base.cc:1034] running on GCE node
I20260808 06:00:27.129021 23720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:27.129791 23720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:27.130970 23720 hybrid_clock.cc:648] HybridClock initialized: now 1786168827130948 us; error 31 us; skew 500 ppm
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168827, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:27.138640 23720 init.cc:377] Logged in from keytab as oryx/127.13.229.67@KRBTEST.COM (short username oryx)
I20260808 06:00:27.139595 23720 webserver.cc:460] Webserver started at http://127.13.229.67:41887/ using document root <none> and password file <none>
I20260808 06:00:27.140214 23720 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:27.140326 23720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:27.144186 23720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260808 06:00:27.146217 23735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:27.147414 23720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:27.147544 23720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "a13761cd472b4273bd07ee2458664290"
format_stamp: "Formatted at 2026-08-08 06:00:26 on dist-test-slave-zkpd"
server_key: "ed4353fb246e35443913b222bd23c774"
server_key_iv: "f73ece22ba6173d0a8cf7a0df5bdad6a"
server_key_version: "encryptionkey@0"
I20260808 06:00:27.148129 23720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:27.168780 23720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:27.171517 23720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:27.171713 23720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:27.172343 23720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:27.173128 23720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:27.173174 23720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:27.173240 23720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:27.173283 23720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:27.182432 23720 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:43291
I20260808 06:00:27.182463 23854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:43291 every 8 connection(s)
I20260808 06:00:27.183365 23720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:27.189599 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23720
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:00:27.194315 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.185090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:57467 (local address 127.13.229.126:34497)
0808 06:00:27.185226 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.185230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.186044 (+ 814us) server_negotiation.cc:407] Connection header received
0808 06:00:27.187285 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.187287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.187335 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.187407 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.188571 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.189092 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.190325 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.190646 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.190764 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.194198 (+ 3434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:57467: BlockingRecv error: recv got EOF from 127.13.229.67:57467 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 06:00:27.194890 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.185460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:57467)
0808 06:00:27.185894 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.185936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.187100 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.187576 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.187582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.187974 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.188452 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.188465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.189198 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.189200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.190189 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.190201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.190837 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.190840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.191011 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.191568 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.191592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.194098 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
W20260808 06:00:27.195391 23855 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:34497 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.229.126:34497: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:27.200117 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.190605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.13.229.126:34497)
0808 06:00:27.190849 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.190853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.190875 (+ 22us) server_negotiation.cc:407] Connection header received
0808 06:00:27.190901 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.190903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.190931 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.190997 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.191979 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.192662 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.193905 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.194133 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.194237 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.195807 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:27.195814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:27.195831 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:27.195862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:27.197836 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:27.198300 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:27.198305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:27.198310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:27.198428 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:27.198728 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:27.198733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:27.198736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:27.199236 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:27.199436 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:27.199666 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 06:00:27.199938 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:27.201167 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.195782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:35535)
0808 06:00:27.195872 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.195884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.195994 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.196240 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.196246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.196413 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.196530 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.196536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.197546 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.197549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.198062 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.198070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.198578 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.198581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.198647 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.199063 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.199077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.201062 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260808 06:00:27.201277 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.195847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35535 (local address 127.13.229.126:34497)
0808 06:00:27.196005 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.196011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.196025 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:27.196058 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.196062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.196102 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.196186 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.196641 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.197437 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.198171 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.198438 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.198555 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.201190 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:35535: BlockingRecv error: recv got EOF from 127.13.229.67:35535 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:27.206188 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.201772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:48389)
0808 06:00:27.201918 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.201928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.201983 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.202191 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.202194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.202350 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.202458 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.202468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.203237 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.203239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.203724 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.203731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.204197 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.204199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.204254 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.204625 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.204635 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.206108 (+ 1473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260808 06:00:27.206355 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.201869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:48389 (local address 127.13.229.126:34497)
0808 06:00:27.202011 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.202015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.202029 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:27.202064 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.202066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.202101 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.202174 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.202576 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.203142 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.203835 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.204105 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.204181 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.206239 (+ 2058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:48389: BlockingRecv error: recv got EOF from 127.13.229.67:48389 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 06:00:27.206401 23855 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:27 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 06:00:27.823319 23861 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:27.818114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:60015 (local address 127.13.229.126:34497)
0808 06:00:27.818347 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:27.818350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:27.818363 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:27.818470 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:27.818473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:27.818517 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:27.818594 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:27.818986 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.819513 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.820330 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.820549 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.820624 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.823196 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:60015: BlockingRecv error: recv got EOF from 127.13.229.65:60015 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260808 06:00:27.823344 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:27.818021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:60015)
0808 06:00:27.818284 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:27.818296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:27.818386 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:27.818629 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:27.818631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:27.818797 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:27.818892 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.818896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.819604 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.819606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.820228 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:27.820238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:27.820642 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:27.820645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:27.820703 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:27.821179 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:27.821190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:27.823034 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:28.021414 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:28.015446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:39309)
0808 06:00:28.015732 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:28.015746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:28.015870 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:28.016167 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:28.016170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:28.016371 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:28.016503 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.016510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.017268 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.017271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.018086 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.018098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.018661 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.018664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.018807 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.019402 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:28.019414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:28.021300 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
W20260808 06:00:28.021559 23861 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:28.015587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:39309 (local address 127.13.229.126:34497)
0808 06:00:28.015750 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:28.015754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:28.015842 (+ 88us) server_negotiation.cc:407] Connection header received
0808 06:00:28.015974 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:28.015976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:28.016021 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:28.016089 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:28.016617 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.017144 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.018223 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.018523 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.018642 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.021437 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:39309: BlockingRecv error: recv got EOF from 127.13.229.66:39309 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:28.213202 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:28.206971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:40481)
0808 06:00:28.207314 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:28.207330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:28.207477 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:28.207845 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:28.207849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:28.208101 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:28.208200 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.208207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.209069 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.209074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.209698 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.209708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.210236 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.210240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.210340 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.211038 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:28.211050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:28.213074 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":83,"threads_started":1}
W20260808 06:00:28.213352 23861 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:28.207104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:40481 (local address 127.13.229.126:34497)
0808 06:00:28.207268 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:28.207272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:28.207420 (+ 148us) server_negotiation.cc:407] Connection header received
0808 06:00:28.207592 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:28.207596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:28.207655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:28.207809 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:28.208345 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.208935 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.209830 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.210079 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.210224 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.213213 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:40481: BlockingRecv error: recv got EOF from 127.13.229.67:40481 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:28 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:28.829747 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:28.824152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:33481)
0808 06:00:28.824462 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:28.824485 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:28.824577 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:28.824838 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:28.824840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:28.825019 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:28.825118 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.825123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.825977 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.825979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.826565 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:28.826574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.827097 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.827099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:28.827162 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.827707 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:28.827718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:28.829611 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
W20260808 06:00:28.829994 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:28.824261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:33481 (local address 127.13.229.126:34497)
0808 06:00:28.824570 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:28.824581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:28.824595 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:28.824635 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:28.824638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:28.824680 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:28.824787 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:28.825232 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.825863 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.826732 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:28.826994 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:28.827079 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:28.829847 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:33481: BlockingRecv error: recv got EOF from 127.13.229.65:33481 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:29.028519 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:29.022257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:51135)
0808 06:00:29.022640 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:29.022655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:29.022762 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:29.023010 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:29.023012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:29.023210 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:29.023328 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.023334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.024094 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.024097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.024723 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.024733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.025174 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.025177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.025263 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.025928 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:29.025941 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:29.028380 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":194,"threads_started":1}
W20260808 06:00:29.028678 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:29.022348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:51135 (local address 127.13.229.126:34497)
0808 06:00:29.022496 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:29.022500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:29.022725 (+ 225us) server_negotiation.cc:407] Connection header received
0808 06:00:29.022842 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:29.022845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:29.022889 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:29.022960 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:29.023440 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.023987 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.024848 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.025060 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.025137 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.028524 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:51135: BlockingRecv error: recv got EOF from 127.13.229.66:51135 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:29.219235 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:29.213975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:37969)
0808 06:00:29.214249 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:29.214263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:29.214371 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:29.214606 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:29.214608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:29.214805 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:29.214903 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.214908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.215620 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.215622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.216209 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.216218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.216619 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.216621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.216681 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.217178 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:29.217190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:29.219098 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
W20260808 06:00:29.219324 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:29.214132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:37969 (local address 127.13.229.126:34497)
0808 06:00:29.214294 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:29.214297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:29.214326 (+ 29us) server_negotiation.cc:407] Connection header received
0808 06:00:29.214455 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:29.214457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:29.214502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:29.214567 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:29.215003 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.215524 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.216303 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.216524 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.216600 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.219234 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:37969: BlockingRecv error: recv got EOF from 127.13.229.67:37969 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:29 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:29.836197 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:29.830524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:47753)
0808 06:00:29.830857 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:29.830871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:29.830974 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:29.831217 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:29.831220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:29.831404 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:29.831505 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.831510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.832245 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.832247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.832793 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:29.832825 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.833231 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.833233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:29.833295 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.833910 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:29.833923 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:29.835999 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
W20260808 06:00:29.836357 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:29.830583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:47753 (local address 127.13.229.126:34497)
0808 06:00:29.830877 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:29.830881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:29.830946 (+ 65us) server_negotiation.cc:407] Connection header received
0808 06:00:29.831054 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:29.831057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:29.831101 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:29.831177 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:29.831603 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.832153 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.832920 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:29.833133 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:29.833209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:29.836176 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:47753: BlockingRecv error: recv got EOF from 127.13.229.65:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:30.035148 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:30.029342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:57907)
0808 06:00:30.029650 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:30.029662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:30.029799 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:30.030080 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:30.030083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:30.030287 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:30.030386 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.030391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.031149 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.031154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.031771 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.031780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.032290 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.032294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.032397 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.032948 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:30.032960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:30.035014 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
W20260808 06:00:30.035300 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:30.029418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:57907 (local address 127.13.229.126:34497)
0808 06:00:30.029590 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:30.029594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:30.029739 (+ 145us) server_negotiation.cc:407] Connection header received
0808 06:00:30.029911 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:30.029914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:30.029967 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:30.030053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:30.030504 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.031038 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.031927 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.032157 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.032285 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.035140 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:57907: BlockingRecv error: recv got EOF from 127.13.229.66:57907 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:30.225956 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:30.220216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:35705)
0808 06:00:30.220484 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:30.220498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:30.220593 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:30.220908 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:30.220911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:30.221137 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:30.221283 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.221289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.222120 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.222126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.222708 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.222716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.223198 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.223201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.223288 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.223858 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:30.223872 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:30.225762 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
W20260808 06:00:30.226017 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:30.220281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35705 (local address 127.13.229.126:34497)
0808 06:00:30.220434 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:30.220438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:30.220568 (+ 130us) server_negotiation.cc:407] Connection header received
0808 06:00:30.220682 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:30.220684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:30.220733 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:30.220821 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:30.221468 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.222003 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.222839 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.223077 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.223185 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.225893 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:35705: BlockingRecv error: recv got EOF from 127.13.229.67:35705 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:30 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:30.842957 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:30.837172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:60529)
0808 06:00:30.837483 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:30.837497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:30.837589 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:30.837872 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:30.837875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:30.838052 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:30.838151 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.838156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.838929 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.838931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.839710 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:30.839721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.840345 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.840349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:30.840446 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.840960 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:30.840973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:30.842841 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
W20260808 06:00:30.843160 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:30.837250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:60529 (local address 127.13.229.126:34497)
0808 06:00:30.837522 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:30.837525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:30.837542 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:00:30.837682 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:30.837684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:30.837729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:30.837830 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:30.838255 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.838837 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.839879 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:30.840222 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:30.840317 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:30.842988 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:60529: BlockingRecv error: recv got EOF from 127.13.229.65:60529 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:31.041688 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:31.035844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:53645)
0808 06:00:31.036198 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:31.036211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:31.036361 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:31.036640 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:31.036643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:31.036893 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:31.037057 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.037065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.037852 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.037857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.038429 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.038438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.038915 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.038918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.039009 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.039541 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:31.039554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:31.041572 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
W20260808 06:00:31.041849 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:31.035966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:53645 (local address 127.13.229.126:34497)
0808 06:00:31.036136 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:31.036142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:31.036297 (+ 155us) server_negotiation.cc:407] Connection header received
0808 06:00:31.036466 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:31.036469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:31.036521 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:31.036615 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:31.037184 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.037736 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.038556 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.038801 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.038909 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.041718 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:53645: BlockingRecv error: recv got EOF from 127.13.229.66:53645 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:31.232573 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:31.226790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:35029)
0808 06:00:31.227119 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:31.227132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:31.227232 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:31.227466 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:31.227468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:31.227649 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:31.227769 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.227775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.228601 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.228604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.229302 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.229316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.229824 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.229831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.229935 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.230498 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:31.230511 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:31.232455 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
W20260808 06:00:31.232693 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:31.226893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35029 (local address 127.13.229.126:34497)
0808 06:00:31.227053 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:31.227058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:31.227202 (+ 144us) server_negotiation.cc:407] Connection header received
0808 06:00:31.227314 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:31.227316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:31.227364 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:31.227423 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:31.227872 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.228502 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.229453 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.229711 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.229803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.232582 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:35029: BlockingRecv error: recv got EOF from 127.13.229.67:35029 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:31 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:31.849367 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:31.843858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:41173)
0808 06:00:31.844172 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:31.844185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:31.844268 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:31.844532 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:31.844534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:31.844721 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:31.844817 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.844822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.845553 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.845556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.846106 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:31.846114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.846537 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.846539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:31.846600 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.847134 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:31.847149 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:31.849247 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":117,"threads_started":1}
W20260808 06:00:31.849543 23877 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:31.843993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:41173 (local address 127.13.229.126:34497)
0808 06:00:31.844225 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:31.844228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:31.844242 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:31.844355 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:31.844358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:31.844412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:31.844496 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:31.844933 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.845459 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.846207 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:31.846440 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:31.846516 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:31.849374 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:41173: BlockingRecv error: recv got EOF from 127.13.229.65:41173 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:32.048436 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:32.042792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:43263)
0808 06:00:32.043073 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:32.043091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:32.043191 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:32.043561 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:32.043565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:32.043759 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:32.043851 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.043857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.044677 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.044683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.045270 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.045278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.045751 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.045755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.045845 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.046324 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:32.046336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:32.048268 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
W20260808 06:00:32.048506 23877 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:32.042909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:43263 (local address 127.13.229.126:34497)
0808 06:00:32.043097 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:32.043102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:32.043148 (+ 46us) server_negotiation.cc:407] Connection header received
0808 06:00:32.043296 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:32.043300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:32.043358 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:32.043452 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:32.044022 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.044563 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.045398 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.045637 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.045747 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.048396 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:43263: BlockingRecv error: recv got EOF from 127.13.229.66:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:32.238716 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:32.233444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:49065)
0808 06:00:32.233703 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:32.233718 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:32.233830 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:32.234086 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:32.234088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:32.234270 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:32.234369 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.234376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.235164 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.235166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.235712 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.235719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.236239 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.236242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.236305 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.236824 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:32.236835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:32.238601 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
W20260808 06:00:32.239013 23877 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:32.233525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:49065 (local address 127.13.229.126:34497)
0808 06:00:32.233691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:32.233695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:32.233805 (+ 110us) server_negotiation.cc:407] Connection header received
0808 06:00:32.233919 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:32.233922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:32.233971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:32.234049 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:32.234477 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.235062 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.235824 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.236133 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.236223 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.238852 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:49065: BlockingRecv error: recv got EOF from 127.13.229.67:49065 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:32 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:32.855772 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:32.850135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:52163)
0808 06:00:32.850416 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:32.850430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:32.850519 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:32.850774 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:32.850776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:32.850961 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:32.851070 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.851075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.851946 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.851949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.852599 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:32.852610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.853119 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.853121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:32.853180 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.853668 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:32.853680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:32.855596 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
W20260808 06:00:32.855984 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:32.850204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:52163 (local address 127.13.229.126:34497)
0808 06:00:32.850469 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:32.850473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:32.850488 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:00:32.850610 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:32.850612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:32.850661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:32.850744 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:32.851183 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.851796 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.852736 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:32.853013 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:32.853104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:32.855761 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:52163: BlockingRecv error: recv got EOF from 127.13.229.65:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:33.054944 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:33.049312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:57107)
0808 06:00:33.049594 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:33.049607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:33.049707 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:33.049944 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:33.049946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:33.050132 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:33.050229 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.050234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.051052 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.051055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.051607 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.051615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.052215 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.052219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.052295 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.052786 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:33.052798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:33.054817 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260808 06:00:33.055099 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:33.049374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:57107 (local address 127.13.229.126:34497)
0808 06:00:33.049508 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:33.049511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:33.049675 (+ 164us) server_negotiation.cc:407] Connection header received
0808 06:00:33.049784 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:33.049786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:33.049831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:33.049900 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:33.050330 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.050890 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.051733 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.052080 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.052171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.054976 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:57107: BlockingRecv error: recv got EOF from 127.13.229.66:57107 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:33.245285 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:33.239491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:60957)
0808 06:00:33.239851 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:33.239871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:33.240033 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:33.240324 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:33.240326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:33.240536 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:33.240631 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.240637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.241422 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.241427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.242076 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.242086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.242578 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.242581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.242672 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.243172 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:33.243183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:33.245159 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":84,"threads_started":1}
W20260808 06:00:33.245419 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:33.239606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:60957 (local address 127.13.229.126:34497)
0808 06:00:33.239812 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:33.239817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:33.239968 (+ 151us) server_negotiation.cc:407] Connection header received
0808 06:00:33.240144 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:33.240147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:33.240202 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:33.240288 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:33.240751 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.241306 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.242207 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.242455 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.242565 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.245291 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:60957: BlockingRecv error: recv got EOF from 127.13.229.67:60957 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:33 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:33.862124 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:33.856669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:50193)
0808 06:00:33.857039 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:33.857052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:33.857151 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:33.857396 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:33.857398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:33.857587 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:33.857685 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.857691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.858431 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.858433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.858991 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:33.858999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.859480 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.859482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:33.859546 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.860100 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:33.860112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:33.861990 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":93,"threads_started":1}
W20260808 06:00:33.862291 23884 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:33.856670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:50193 (local address 127.13.229.126:34497)
0808 06:00:33.856979 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:33.856983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:33.857104 (+ 121us) server_negotiation.cc:407] Connection header received
0808 06:00:33.857232 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:33.857234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:33.857281 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:33.857358 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:33.857781 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.858330 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.859090 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:33.859376 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:33.859464 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:33.862147 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:50193: BlockingRecv error: recv got EOF from 127.13.229.65:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:34.061237 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:34.055688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:42329)
0808 06:00:34.056054 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:34.056068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:34.056202 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:34.056458 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:34.056461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:34.056678 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:34.056775 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.056781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.057617 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.057619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.058161 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.058169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.058570 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.058573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.058632 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.059176 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:34.059188 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:34.061123 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":163,"threads_started":1}
W20260808 06:00:34.061530 23884 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:34.055777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:42329 (local address 127.13.229.126:34497)
0808 06:00:34.055972 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:34.055977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:34.056139 (+ 162us) server_negotiation.cc:407] Connection header received
0808 06:00:34.056308 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:34.056310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:34.056355 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:34.056421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:34.056874 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.057508 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.058257 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.058477 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.058549 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.061271 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:42329: BlockingRecv error: recv got EOF from 127.13.229.66:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:34.251485 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:34.246194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:60281)
0808 06:00:34.246513 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:34.246525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:34.246626 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:34.246853 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:34.246855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:34.247043 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:34.247149 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.247154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.247913 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.247916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.248486 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.248493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.248914 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.248917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.248975 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.249454 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:34.249465 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:34.251370 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":81,"threads_started":1}
W20260808 06:00:34.251722 23884 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:34.246278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:60281 (local address 127.13.229.126:34497)
0808 06:00:34.246411 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:34.246415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:34.246630 (+ 215us) server_negotiation.cc:407] Connection header received
0808 06:00:34.246693 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:34.246696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:34.246746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:34.246815 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:34.247249 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.247786 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.248588 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.248815 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.248897 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.251592 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:60281: BlockingRecv error: recv got EOF from 127.13.229.67:60281 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:34 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:34.868645 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:34.862885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:49865)
0808 06:00:34.863171 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:34.863184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:34.863272 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:34.863502 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:34.863504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:34.863687 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:34.863785 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.863790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.864741 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.864744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.865498 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:34.865513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.866057 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.866059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:34.866124 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.866642 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:34.866654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:34.868532 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
W20260808 06:00:34.868835 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:34.863050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:49865 (local address 127.13.229.126:34497)
0808 06:00:34.863285 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:34.863288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:34.863302 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:34.863342 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:34.863344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:34.863386 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:34.863463 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:34.863929 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.864598 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.865650 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:34.865938 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:34.866029 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:34.868686 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:49865: BlockingRecv error: recv got EOF from 127.13.229.65:49865 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:35.067670 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:35.062084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:43935)
0808 06:00:35.062410 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:35.062424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:35.062536 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:35.062760 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:35.062762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:35.062944 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:35.063041 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.063047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.063994 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.063996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.064543 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.064551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.065023 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.065025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.065088 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.065574 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:35.065585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:35.067557 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20260808 06:00:35.067929 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:35.062162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:43935 (local address 127.13.229.126:34497)
0808 06:00:35.062311 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:35.062318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:35.062496 (+ 178us) server_negotiation.cc:407] Connection header received
0808 06:00:35.062606 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:35.062608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:35.062655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:35.062765 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:35.063145 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.063839 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.064658 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.064917 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.065001 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.067731 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:43935: BlockingRecv error: recv got EOF from 127.13.229.66:43935 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:35.258219 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:35.252431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:60635)
0808 06:00:35.252699 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:35.252713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:35.252860 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:35.253101 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:35.253103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:35.253286 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:35.253383 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.253388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.254102 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.254104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.254662 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.254670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.255207 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.255210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.255277 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.255842 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:35.255857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:35.258048 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
W20260808 06:00:35.258281 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:35.252484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:60635 (local address 127.13.229.126:34497)
0808 06:00:35.252629 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:35.252633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:35.252786 (+ 153us) server_negotiation.cc:407] Connection header received
0808 06:00:35.252943 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:35.252946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:35.252995 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:35.253063 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:35.253478 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.254001 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.254769 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.255036 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.255149 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.258172 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:60635: BlockingRecv error: recv got EOF from 127.13.229.67:60635 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:35 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:35.875173 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:35.869589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:35309)
0808 06:00:35.869866 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:35.869881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:35.869978 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:35.870297 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:35.870299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:35.870527 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:35.870625 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.870630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.871422 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.871424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.872056 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:35.872065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.872520 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.872523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:35.872582 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.873159 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:35.873171 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:35.875037 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
W20260808 06:00:35.875363 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:35.869677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:35309 (local address 127.13.229.126:34497)
0808 06:00:35.869917 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:35.869920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:35.869934 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:35.870108 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:35.870111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:35.870175 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:35.870258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:35.870760 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.871315 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.872172 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:35.872419 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:35.872505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:35.875182 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:35309: BlockingRecv error: recv got EOF from 127.13.229.65:35309 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:36.074498 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:36.068541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:42711)
0808 06:00:36.068934 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:36.068952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:36.069087 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:36.069405 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:36.069408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:36.069650 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:36.069793 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.069800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.070638 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.070644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.071240 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.071256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.071749 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.071754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.071837 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.072398 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:36.072411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:36.074386 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
W20260808 06:00:36.074684 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:36.068620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:42711 (local address 127.13.229.126:34497)
0808 06:00:36.068854 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:36.068859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:36.069051 (+ 192us) server_negotiation.cc:407] Connection header received
0808 06:00:36.069186 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:36.069192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:36.069252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:36.069355 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:36.069915 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.070525 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.071402 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.071631 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.071753 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.074531 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:42711: BlockingRecv error: recv got EOF from 127.13.229.66:42711 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:36.265277 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:36.258970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:41627)
0808 06:00:36.259273 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:36.259287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:36.259393 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:36.259741 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:36.259743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:36.260022 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:36.260141 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.260148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.261032 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.261039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.261586 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.261594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.262033 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.262035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.262096 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.262735 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:36.262755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:36.265088 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
W20260808 06:00:36.265336 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:36.259051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:41627 (local address 127.13.229.126:34497)
0808 06:00:36.259215 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:36.259219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:36.259365 (+ 146us) server_negotiation.cc:407] Connection header received
0808 06:00:36.259501 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:36.259505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:36.259570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:36.259696 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:36.260265 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.260912 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.261694 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.261930 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.262013 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.265213 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:41627: BlockingRecv error: recv got EOF from 127.13.229.67:41627 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:36 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:36.881650 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:36.876049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:44895)
0808 06:00:36.876340 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:36.876355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:36.876441 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:36.876692 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:36.876694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:36.876879 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:36.876978 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.876983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.877703 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.877705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.878263 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:36.878280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.878690 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.878692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:36.878757 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.879354 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:36.879367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:36.881475 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
W20260808 06:00:36.881718 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:36.876144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:44895 (local address 127.13.229.126:34497)
0808 06:00:36.876391 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:36.876395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:36.876408 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:36.876530 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:36.876533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:36.876578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:36.876656 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:36.877079 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.877606 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.878374 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:36.878590 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:36.878665 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:36.881597 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:44895: BlockingRecv error: recv got EOF from 127.13.229.65:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:37.080844 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:37.075245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:47461)
0808 06:00:37.075595 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:37.075611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:37.075762 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:37.076069 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:37.076072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:37.076304 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:37.076404 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.076410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.077181 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.077185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.077762 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.077770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.078250 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.078255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.078350 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.078835 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:37.078846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:37.080636 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
W20260808 06:00:37.080940 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:37.075368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:47461 (local address 127.13.229.126:34497)
0808 06:00:37.075541 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:37.075546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:37.075706 (+ 160us) server_negotiation.cc:407] Connection header received
0808 06:00:37.075867 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:37.075870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:37.075953 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:37.076041 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:37.076526 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.077064 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.077890 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.078132 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.078243 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.080813 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:47461: BlockingRecv error: recv got EOF from 127.13.229.66:47461 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:37.276082 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:37.266129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:48485)
0808 06:00:37.266407 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:37.266421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:37.266504 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:37.266746 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:37.266748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:37.266925 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:37.267018 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.267022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.267741 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.267743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.268510 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.268525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.269086 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.269093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.269173 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.274073 (+ 4900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:37.274085 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:37.275958 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
W20260808 06:00:37.276193 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:37.266285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:48485 (local address 127.13.229.126:34497)
0808 06:00:37.266448 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:37.266452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:37.266466 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:37.266592 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:37.266594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:37.266641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:37.266706 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:37.267124 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.267651 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.268660 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.268929 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.269056 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.276072 (+ 7016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:48485: BlockingRecv error: recv got EOF from 127.13.229.67:48485 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:37 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:37.888309 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:37.882483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:37359)
0808 06:00:37.882767 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:37.882787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:37.882893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:37.883128 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:37.883130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:37.883314 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:37.883412 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.883417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.884373 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.884375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.884928 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:37.884936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.885421 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.885423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:37.885484 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.886103 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:37.886115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:37.888133 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
W20260808 06:00:37.888410 23901 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:37.882621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:37359 (local address 127.13.229.126:34497)
0808 06:00:37.882893 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:37.882897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:37.882910 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:37.882965 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:37.882967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:37.883011 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:37.883087 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:37.883555 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.884267 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.885052 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:37.885314 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:37.885406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:37.888263 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:37359: BlockingRecv error: recv got EOF from 127.13.229.65:37359 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:38.087297 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:38.081704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:52111)
0808 06:00:38.082033 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:38.082048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:38.082142 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:38.082413 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:38.082416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:38.082604 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:38.082702 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.082707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.083490 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.083492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.084139 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.084149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.084612 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.084614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.084675 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.085248 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:38.085260 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:38.087183 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
W20260808 06:00:38.087488 23901 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:38.081901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:52111 (local address 127.13.229.126:34497)
0808 06:00:38.082080 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:38.082084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:38.082100 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:38.082250 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:38.082252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:38.082301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:38.082377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:38.082814 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.083395 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.084270 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.084511 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.084599 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.087345 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:52111: BlockingRecv error: recv got EOF from 127.13.229.66:52111 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:38.282379 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:38.276879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:41059)
0808 06:00:38.277160 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:38.277175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:38.277284 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:38.277601 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:38.277604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:38.277799 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:38.277910 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.277916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.278672 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.278675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.279236 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.279245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.279668 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.279670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.279732 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.280348 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:38.280360 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:38.282257 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
W20260808 06:00:38.282518 23901 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:38.276955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:41059 (local address 127.13.229.126:34497)
0808 06:00:38.277120 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:38.277124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:38.277257 (+ 133us) server_negotiation.cc:407] Connection header received
0808 06:00:38.277389 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:38.277391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:38.277442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:38.277523 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:38.278025 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.278558 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.279347 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.279568 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.279647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.282396 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:41059: BlockingRecv error: recv got EOF from 127.13.229.67:41059 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:38 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:38.894829 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:38.889140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:54959)
0808 06:00:38.889419 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:38.889433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:38.889522 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:38.889755 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:38.889757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:38.889954 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:38.890055 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.890060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.890805 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.890808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.891358 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:38.891365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.891810 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.891813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:38.891939 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.892478 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:38.892491 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:38.894706 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
W20260808 06:00:38.894982 23905 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:38.889294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:54959 (local address 127.13.229.126:34497)
0808 06:00:38.889529 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:38.889533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:38.889546 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:38.889599 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:38.889601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:38.889644 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:38.889716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:38.890171 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.890709 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.891460 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:38.891712 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:38.891814 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:38.894833 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:54959: BlockingRecv error: recv got EOF from 127.13.229.65:54959 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:39.093508 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:39.088151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:47261)
0808 06:00:39.088435 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:39.088449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:39.088545 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:39.088758 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:39.088760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:39.088941 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:39.089038 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.089043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.089896 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.089898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.090456 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.090462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.090947 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.090949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.091009 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.091527 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:39.091539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:39.093396 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
W20260808 06:00:39.093694 23905 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:39.088220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:47261 (local address 127.13.229.126:34497)
0808 06:00:39.088418 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:39.088422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:39.088499 (+ 77us) server_negotiation.cc:407] Connection header received
0808 06:00:39.088610 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:39.088612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:39.088655 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:39.088720 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:39.089174 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.089789 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.090576 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.090842 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.090931 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.093572 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:47261: BlockingRecv error: recv got EOF from 127.13.229.66:47261 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:39.288910 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:39.283316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:40241)
0808 06:00:39.283671 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:39.283684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:39.283826 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:39.284138 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:39.284141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:39.284355 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:39.284456 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.284462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.285232 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.285237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.285795 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.285803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.286282 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.286286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.286372 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.286914 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:39.286925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:39.288793 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
W20260808 06:00:39.289103 23905 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:39.283431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:40241 (local address 127.13.229.126:34497)
0808 06:00:39.283616 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:39.283621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:39.283764 (+ 143us) server_negotiation.cc:407] Connection header received
0808 06:00:39.283952 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:39.283957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:39.284013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:39.284109 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:39.284577 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.285115 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.285923 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.286155 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.286267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.288967 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:40241: BlockingRecv error: recv got EOF from 127.13.229.67:40241 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:39 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:39.901350 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:39.895652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:56579)
0808 06:00:39.895954 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:39.895972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:39.896081 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:39.896401 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:39.896405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:39.896620 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:39.896780 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.896789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.897708 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.897711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.898291 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:39.898299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.898723 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.898725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:39.898789 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.899293 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:39.899305 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:39.901229 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
W20260808 06:00:39.901536 23909 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:39.895763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:56579 (local address 127.13.229.126:34497)
0808 06:00:39.896074 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:39.896079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:39.896095 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:00:39.896143 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:39.896149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:39.896215 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:39.896343 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:39.896919 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.897603 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.898402 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:39.898625 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:39.898703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:39.901368 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:56579: BlockingRecv error: recv got EOF from 127.13.229.65:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:40.099938 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:40.094328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:41551)
0808 06:00:40.094680 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:40.094696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:40.094839 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:40.095115 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:40.095118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:40.095338 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:40.095440 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.095445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.096284 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.096290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.096853 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.096861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.097323 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.097327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.097422 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.097937 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:40.097948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:40.099739 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":152,"threads_started":1}
W20260808 06:00:40.100031 23909 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:40.094504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:41551 (local address 127.13.229.126:34497)
0808 06:00:40.094660 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:40.094665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:40.094776 (+ 111us) server_negotiation.cc:407] Connection header received
0808 06:00:40.094945 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:40.094949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:40.095004 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:40.095089 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:40.095559 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.096175 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.096979 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.097208 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.097318 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.099912 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:41551: BlockingRecv error: recv got EOF from 127.13.229.66:41551 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:40.295202 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:40.289757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:50267)
0808 06:00:40.290088 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:40.290101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:40.290236 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:40.290511 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:40.290514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:40.290732 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:40.290827 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.290833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.291591 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.291595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.292194 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.292203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.292666 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.292670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.292759 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.293251 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:40.293262 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:40.295033 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
W20260808 06:00:40.295272 23909 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:40.289833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:50267 (local address 127.13.229.126:34497)
0808 06:00:40.290013 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:40.290017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:40.290174 (+ 157us) server_negotiation.cc:407] Connection header received
0808 06:00:40.290335 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:40.290339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:40.290396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:40.290483 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:40.290949 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.291479 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.292317 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.292549 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.292662 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.295156 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:50267: BlockingRecv error: recv got EOF from 127.13.229.67:50267 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:40 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:40.908030 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:40.902159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:37975)
0808 06:00:40.902450 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:40.902463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:40.902555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:40.902772 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:40.902774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:40.902956 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:40.903066 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.903071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.903872 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.903874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.904860 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:40.904873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.905497 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.905501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:40.905590 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.906105 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:40.906117 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:40.907884 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
W20260808 06:00:40.908253 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:40.902231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:37975 (local address 127.13.229.126:34497)
0808 06:00:40.902557 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:40.902560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:40.902574 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:40.902615 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:40.902617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:40.902659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:40.902734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:40.903216 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.903766 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.905001 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:40.905370 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:40.905477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:40.908093 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:37975: BlockingRecv error: recv got EOF from 127.13.229.65:37975 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:41.106119 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:41.100794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:39441)
0808 06:00:41.101090 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:41.101103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:41.101204 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:41.101426 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:41.101428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:41.101612 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:41.101709 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.101714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.102567 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.102570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.103125 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.103133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.103556 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.103559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.103619 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.104162 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:41.104173 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:41.106007 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":119,"threads_started":1}
W20260808 06:00:41.106307 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:41.100833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:39441 (local address 127.13.229.126:34497)
0808 06:00:41.100984 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:41.100989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:41.101175 (+ 186us) server_negotiation.cc:407] Connection header received
0808 06:00:41.101270 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:41.101273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:41.101318 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:41.101398 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:41.101805 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.102468 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.103230 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.103448 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.103524 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.106146 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:39441: BlockingRecv error: recv got EOF from 127.13.229.66:39441 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.67@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:41.301533 23915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:41.296143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.67:57401)
0808 06:00:41.296422 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:41.296437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:41.296581 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:41.296911 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:41.296913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:41.297102 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:41.297198 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.297203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.297943 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.297945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.298523 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.298531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.298961 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.298964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.299022 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.299503 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:41.299515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:41.301407 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20260808 06:00:41.301699 23913 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:41.296196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:57401 (local address 127.13.229.126:34497)
0808 06:00:41.296358 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:41.296364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:41.296546 (+ 182us) server_negotiation.cc:407] Connection header received
0808 06:00:41.296691 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:41.296695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:41.296758 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:41.296869 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:41.297301 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.297848 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.298622 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.298859 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.298940 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.301543 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.67:57401: BlockingRecv error: recv got EOF from 127.13.229.67:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:41 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.65@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:41.914474 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:41.908909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.65:48653)
0808 06:00:41.909210 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:41.909224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:41.909308 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:41.909540 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:41.909542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:41.909725 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:41.909823 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.909828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.910591 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.910593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.911164 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:41.911172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.911587 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.911590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:41.911652 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.912156 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:41.912168 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:41.914301 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260808 06:00:41.914532 23917 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:41.909041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48653 (local address 127.13.229.126:34497)
0808 06:00:41.909280 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:41.909283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:41.909297 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:00:41.909390 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:41.909392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:41.909434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:41.909505 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:41.909945 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.910496 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.911269 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:41.911491 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:41.911568 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:41.914411 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.65:48653: BlockingRecv error: recv got EOF from 127.13.229.65:48653 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
Aug 08 06:00:42 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 06:00:42 dist-test-slave-zkpd krb5kdc[22813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.229.66@KRBTEST.COM for oryx/127.13.229.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 06:00:42.112809 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.107053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34497 (local address 127.13.229.66:44785)
0808 06:00:42.107325 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:42.107338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:42.107425 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:42.107782 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:42.107785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:42.108075 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:42.108205 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.108214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.109112 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.109115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.109661 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.109669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.110088 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.110090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.110154 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.110628 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:42.110653 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:42.112611 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.229.126:34497: Server oryx/127.13.229.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
W20260808 06:00:42.112838 23917 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:42.107173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:44785 (local address 127.13.229.126:34497)
0808 06:00:42.107349 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:42.107353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:42.107387 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:00:42.107541 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:42.107544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:42.107627 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:42.107712 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:42.108346 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.109013 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.109764 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.109987 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.110068 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.112734 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.229.66:44785: BlockingRecv error: recv got EOF from 127.13.229.66:44785 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 06:00:42.198385 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23421
I20260808 06:00:42.206305 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23580
I20260808 06:00:42.213382 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23720
I20260808 06:00:42.220119 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23349
2026-08-08T06:00:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18049 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T06:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:42Z Disabled control of system clock
I20260808 06:00:42.254591 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37885
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 06:00:42.374788 23922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:42.375049 23922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:42.375083 23922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:42.379021 23922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:42.379071 23922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:42.379086 23922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:42.379102 23922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:42.379117 23922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:42.384465 23922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:37885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:42.385476 23922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:42.386795 23922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:42.395278 23928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:42.395293 23927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:42.395386 23922 server_base.cc:1034] running on GCE node
W20260808 06:00:42.395298 23930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:42.396044 23922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:42.397107 23922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:42.398305 23922 hybrid_clock.cc:648] HybridClock initialized: now 1786168842398290 us; error 35 us; skew 500 ppm
I20260808 06:00:42.404343 23922 webserver.cc:460] Webserver started at http://127.13.229.126:40039/ using document root <none> and password file <none>
I20260808 06:00:42.404925 23922 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:42.405020 23922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:42.405256 23922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:42.407279 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "b564c291d939458bbb5fe922e37916ec"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "5eb44677688f94985bfb13dfdc47d034"
server_key_iv: "0bbf04ad766e30c88cf0cb3351d808cd"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.407848 23922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "b564c291d939458bbb5fe922e37916ec"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "5eb44677688f94985bfb13dfdc47d034"
server_key_iv: "0bbf04ad766e30c88cf0cb3351d808cd"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.411417 23922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:00:42.413786 23938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.414940 23922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:42.415078 23922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "b564c291d939458bbb5fe922e37916ec"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "5eb44677688f94985bfb13dfdc47d034"
server_key_iv: "0bbf04ad766e30c88cf0cb3351d808cd"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.415184 23922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:42.430598 23922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:42.431227 23922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:42.431425 23922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:42.438510 23922 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:37885
I20260808 06:00:42.438515 23992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:37885 every 8 connection(s)
I20260808 06:00:42.439399 23922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:42.441129 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 23922
I20260808 06:00:42.441268 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:42.441556 14229 external_mini_cluster.cc:1428] Setting key 749e6c5d42a5beb271d139f5f66dfa1e
I20260808 06:00:42.442946 23993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:42.449635 23993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec: Bootstrap starting.
I20260808 06:00:42.451601 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.442864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.13.229.126:37885)
0808 06:00:42.443524 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:42.443567 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:42.443609 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:00:42.444505 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:42.444526 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:42.445017 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:42.445371 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:42.445808 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.447252 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.448171 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.448892 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.449048 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.449971 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:42.449976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:42.449988 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:42.450360 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:42.450413 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:42.450433 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:42.450542 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:42.450679 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 06:00:42.450925 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":144,"threads_started":1}
I20260808 06:00:42.451879 23993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:42.452769 23993 log.cc:824] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:42.454660 23993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec: No bootstrap required, opened a new log
I20260808 06:00:42.458171 23993 raft_consensus.cc:348] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } }
I20260808 06:00:42.458436 23993 raft_consensus.cc:374] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:42.458498 23993 raft_consensus.cc:729] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b564c291d939458bbb5fe922e37916ec, State: Initialized, Role: FOLLOWER
I20260808 06:00:42.458985 23993 consensus_queue.cc:260] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } }
I20260808 06:00:42.459144 23993 raft_consensus.cc:388] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:42.459270 23993 raft_consensus.cc:482] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:42.459385 23993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:42.460606 23993 raft_consensus.cc:504] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } }
I20260808 06:00:42.460968 23993 leader_election.cc:302] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b564c291d939458bbb5fe922e37916ec; no voters:
I20260808 06:00:42.461289 23993 leader_election.cc:288] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:42.461375 23999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:42.461601 23999 raft_consensus.cc:686] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [term 1 LEADER]: Becoming Leader. State: Replica: b564c291d939458bbb5fe922e37916ec, State: Running, Role: LEADER
I20260808 06:00:42.461943 23999 consensus_queue.cc:237] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } }
I20260808 06:00:42.462409 23993 sys_catalog.cc:564] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:42.463594 24001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b564c291d939458bbb5fe922e37916ec. Latest consensus state: current_term: 1 leader_uuid: "b564c291d939458bbb5fe922e37916ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } } }
I20260808 06:00:42.463594 24000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b564c291d939458bbb5fe922e37916ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b564c291d939458bbb5fe922e37916ec" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37885 } } }
I20260808 06:00:42.463784 24001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:42.463816 24000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:42.464771 24009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:42.467685 24009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:42.473423 24009 catalog_manager.cc:1347] Generated new cluster ID: abef4d3ac2f14c0c97bfb7fbe25d79bf
I20260808 06:00:42.473546 24009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:42.484851 24009 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:42.485980 24009 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:42.499511 24009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b564c291d939458bbb5fe922e37916ec: Generated new TSK 0
I20260808 06:00:42.500269 24009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:42.508468 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--builtin_ntp_servers=127.13.229.84:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 06:00:42.620354 24019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:42.620601 24019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:42.620636 24019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:42.624665 24019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:42.624722 24019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:42.624778 24019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:42.629431 24019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:42.630491 24019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:42.631939 24019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:42.641249 24025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:42.641245 24024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:42.641366 24027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:42.641449 24019 server_base.cc:1034] running on GCE node
I20260808 06:00:42.641878 24019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:42.642521 24019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:42.643712 24019 hybrid_clock.cc:648] HybridClock initialized: now 1786168842643693 us; error 28 us; skew 500 ppm
I20260808 06:00:42.645540 24019 webserver.cc:460] Webserver started at http://127.13.229.65:35649/ using document root <none> and password file <none>
I20260808 06:00:42.646092 24019 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:42.646186 24019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:42.646402 24019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:42.648445 24019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "baeb1450c75348d48c526406868521f8"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "42821dab6bd45f52559bbd014c18499f"
server_key_iv: "13e03efa1ecbea4858d5d6f83a295506"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.649004 24019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "baeb1450c75348d48c526406868521f8"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "42821dab6bd45f52559bbd014c18499f"
server_key_iv: "13e03efa1ecbea4858d5d6f83a295506"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.652485 24019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:42.654876 24033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.656107 24019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:42.656236 24019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "baeb1450c75348d48c526406868521f8"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "42821dab6bd45f52559bbd014c18499f"
server_key_iv: "13e03efa1ecbea4858d5d6f83a295506"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.656361 24019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:42.678862 24019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:42.679580 24019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:42.679783 24019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:42.680438 24019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:42.681401 24019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:42.681474 24019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.681545 24019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:42.681596 24019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.690179 24019 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:35145
I20260808 06:00:42.690212 24152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:35145 every 8 connection(s)
I20260808 06:00:42.691064 24019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:42.695431 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24019
I20260808 06:00:42.695557 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:42.695866 14229 external_mini_cluster.cc:1428] Setting key 68a8378141fe75787fb1972b663263b5
I20260808 06:00:42.698046 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--builtin_ntp_servers=127.13.229.84:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 06:00:42.701282 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.692831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:38049 (local address 127.13.229.126:37885)
0808 06:00:42.692988 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:42.692993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:42.693958 (+ 965us) server_negotiation.cc:407] Connection header received
0808 06:00:42.695377 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:42.695381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:42.695449 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:42.695536 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:42.696740 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.697492 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.699196 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.699425 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.699507 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.700780 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:42.700784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:42.700786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:42.700798 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:42.700817 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:42.700825 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:42.700904 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:42.701069 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 06:00:42.701143 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 06:00:42.702082 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.693147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37885 (local address 127.13.229.65:38049)
0808 06:00:42.693806 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:42.693845 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:42.695149 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:42.695769 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:42.695779 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:42.695885 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:42.696569 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.696591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.697629 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.697632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.699065 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.699080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.699580 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.699587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.699819 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.700515 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:42.700516 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:42.700554 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:42.700614 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:42.700918 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:42.700929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:42.700935 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:00:42.701066 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 06:00:42.701341 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260808 06:00:42.703198 24153 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37885
I20260808 06:00:42.703428 24153 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:42.703989 24153 heartbeater.cc:511] Master 127.13.229.126:37885 requested a full tablet report, sending...
I20260808 06:00:42.705415 23956 ts_manager.cc:194] Registered new tserver with Master: baeb1450c75348d48c526406868521f8 (127.13.229.65:35145)
I20260808 06:00:42.706511 23956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.65:38049
W20260808 06:00:42.808050 24156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:42.808322 24156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:42.808382 24156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:42.812345 24156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:42.812413 24156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:42.812492 24156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:42.817132 24156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:42.818143 24156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:42.819591 24156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:42.828878 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:42.828866 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:42.829032 24156 server_base.cc:1034] running on GCE node
W20260808 06:00:42.829223 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:42.829656 24156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:42.830288 24156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:42.831446 24156 hybrid_clock.cc:648] HybridClock initialized: now 1786168842831433 us; error 25 us; skew 500 ppm
I20260808 06:00:42.833278 24156 webserver.cc:460] Webserver started at http://127.13.229.66:35635/ using document root <none> and password file <none>
I20260808 06:00:42.833858 24156 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:42.833923 24156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:42.834216 24156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:42.836295 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "cf72b9524c1940d69c64180939ddd38b"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "7a982f694b4628d67d89973eac03580f"
server_key_iv: "efa07966a88ef202c0c537945fea6fc3"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.836869 24156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "cf72b9524c1940d69c64180939ddd38b"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "7a982f694b4628d67d89973eac03580f"
server_key_iv: "efa07966a88ef202c0c537945fea6fc3"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.840507 24156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:42.842921 24171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.844153 24156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:42.844301 24156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "cf72b9524c1940d69c64180939ddd38b"
format_stamp: "Formatted at 2026-08-08 06:00:42 on dist-test-slave-zkpd"
server_key: "7a982f694b4628d67d89973eac03580f"
server_key_iv: "efa07966a88ef202c0c537945fea6fc3"
server_key_version: "encryptionkey@0"
I20260808 06:00:42.844463 24156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:42.864483 24156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:42.865156 24156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:42.865351 24156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:42.865934 24156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:42.866875 24156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:42.866947 24156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.867015 24156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:42.867057 24156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:42.876124 24156 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:36679
I20260808 06:00:42.876157 24290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:36679 every 8 connection(s)
I20260808 06:00:42.876986 24156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:42.885149 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24156
I20260808 06:00:42.885241 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:42.885586 14229 external_mini_cluster.cc:1428] Setting key 50b20543616c02fc57a3bd1486297225
I20260808 06:00:42.886293 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.878728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:48941 (local address 127.13.229.126:37885)
0808 06:00:42.878917 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:42.878924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:42.880312 (+ 1388us) server_negotiation.cc:407] Connection header received
0808 06:00:42.881711 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:42.881713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:42.881760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:42.881832 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:42.882639 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.883143 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.884302 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.884510 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.884589 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.885767 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:42.885770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:42.885772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:42.885784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:42.885803 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:42.885810 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:42.885877 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:42.886066 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 06:00:42.886134 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 06:00:42.887224 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:42.879235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37885 (local address 127.13.229.66:48941)
0808 06:00:42.880177 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:42.880218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:42.881528 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:42.881961 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:42.881968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:42.882047 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:42.882518 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.882531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.883253 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.883255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.884201 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:42.884210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:42.884639 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:42.884642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:42.884787 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:42.885454 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:42.885456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:42.885512 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:42.885589 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:42.885903 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:42.885913 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:42.885919 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:00:42.886052 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 06:00:42.886317 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":164,"threads_started":1}
I20260808 06:00:42.887864 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--builtin_ntp_servers=127.13.229.84:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 06:00:42.888334 24292 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37885
I20260808 06:00:42.888573 24292 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:42.889060 24292 heartbeater.cc:511] Master 127.13.229.126:37885 requested a full tablet report, sending...
I20260808 06:00:42.890096 23956 ts_manager.cc:194] Registered new tserver with Master: cf72b9524c1940d69c64180939ddd38b (127.13.229.66:36679)
I20260808 06:00:42.890889 23956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.66:48941
W20260808 06:00:43.003224 24295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:43.003492 24295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:43.003568 24295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:43.007611 24295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:43.007689 24295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:43.007774 24295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:43.012449 24295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:43.013463 24295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:43.014865 24295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:43.024509 24295 server_base.cc:1034] running on GCE node
W20260808 06:00:43.024446 24300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.024475 24301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.024449 24303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.025149 24295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:43.025759 24295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:43.026965 24295 hybrid_clock.cc:648] HybridClock initialized: now 1786168843026930 us; error 46 us; skew 500 ppm
I20260808 06:00:43.028751 24295 webserver.cc:460] Webserver started at http://127.13.229.67:36941/ using document root <none> and password file <none>
I20260808 06:00:43.029340 24295 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:43.029430 24295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:43.029659 24295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:43.031628 24295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "7486e6d3bd484d5585db13f032079c86"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "c6e764260953e8d206d0b28b6b216470"
server_key_iv: "9f75c9a8b72c7d67952aee5c629d37e0"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.032231 24295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "7486e6d3bd484d5585db13f032079c86"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "c6e764260953e8d206d0b28b6b216470"
server_key_iv: "9f75c9a8b72c7d67952aee5c629d37e0"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.035702 24295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:00:43.038192 24309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.039427 24295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:43.039557 24295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "7486e6d3bd484d5585db13f032079c86"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "c6e764260953e8d206d0b28b6b216470"
server_key_iv: "9f75c9a8b72c7d67952aee5c629d37e0"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.039674 24295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:43.069996 24295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:43.070719 24295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:43.070919 24295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:43.071523 24295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:43.072533 24295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:43.072605 24295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.072682 24295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:43.072724 24295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.081532 24295 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:35111
I20260808 06:00:43.081569 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:35111 every 8 connection(s)
I20260808 06:00:43.082482 24295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:43.085585 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24295
I20260808 06:00:43.085702 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:43.085942 14229 external_mini_cluster.cc:1428] Setting key eccd4e0c2379c2f82cfa98a1410b4e5a
I20260808 06:00:43.091812 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.084412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:59587 (local address 127.13.229.126:37885)
0808 06:00:43.084568 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.084572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.085654 (+ 1082us) server_negotiation.cc:407] Connection header received
0808 06:00:43.087266 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.087269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.087341 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.087428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.088553 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:43.089071 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:43.090147 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:43.090372 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:43.090447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:43.091419 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:43.091422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:43.091424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:43.091435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:43.091453 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:43.091461 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:43.091520 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:43.091654 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 06:00:43.091702 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 06:00:43.092643 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.084803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37885 (local address 127.13.229.67:59587)
0808 06:00:43.085494 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:43.085534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:43.087032 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:43.087637 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:43.087645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:43.087748 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:43.088426 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:43.088441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:43.089174 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:43.089176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:43.090043 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:43.090052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:43.090493 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:43.090495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:43.090626 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:43.091183 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:43.091185 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:43.091224 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:43.091284 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:43.091535 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:43.091541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:43.091545 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:00:43.091636 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 06:00:43.091843 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260808 06:00:43.093758 24430 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37885
I20260808 06:00:43.093950 24430 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:43.094419 24430 heartbeater.cc:511] Master 127.13.229.126:37885 requested a full tablet report, sending...
I20260808 06:00:43.095350 23956 ts_manager.cc:194] Registered new tserver with Master: 7486e6d3bd484d5585db13f032079c86 (127.13.229.67:35111)
I20260808 06:00:43.095865 23956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.67:59587
I20260808 06:00:43.099845 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 06:00:43.102639 23997 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:43.101667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.13.229.126:37885)
0808 06:00:43.101807 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.101812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.102041 (+ 229us) server_negotiation.cc:407] Connection header received
0808 06:00:43.102123 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.102127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.102181 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.102285 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.102522 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47452: BlockingRecv error: recv got EOF from 127.0.0.1:47452 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:00:43.102993 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24019
I20260808 06:00:43.113997 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24156
I20260808 06:00:43.120406 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24295
I20260808 06:00:43.126631 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 23922
2026-08-08T06:00:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (905 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T06:00:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:43Z Disabled control of system clock
I20260808 06:00:43.157884 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45801
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 06:00:43.268586 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:43.268826 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:43.268863 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:43.272802 24442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:43.272866 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:43.272881 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:43.272898 24442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:43.272913 24442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:43.277464 24442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:43.278425 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:43.279693 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:43.288105 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.288089 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.288086 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.288414 24442 server_base.cc:1034] running on GCE node
I20260808 06:00:43.289067 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:43.290138 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:43.291356 24442 hybrid_clock.cc:648] HybridClock initialized: now 1786168843291341 us; error 37 us; skew 500 ppm
I20260808 06:00:43.293430 24442 webserver.cc:460] Webserver started at http://127.13.229.126:33301/ using document root <none> and password file <none>
I20260808 06:00:43.294046 24442 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:43.294144 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:43.294381 24442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:43.296597 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "8570d02d8899451bb4492759213ce62d"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "fd60e7e6a1cbd08129d074438b65e3bb"
server_key_iv: "39e6e030edec97ade3805e211fc2eb34"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.297189 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "8570d02d8899451bb4492759213ce62d"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "fd60e7e6a1cbd08129d074438b65e3bb"
server_key_iv: "39e6e030edec97ade3805e211fc2eb34"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.300916 24442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:43.303237 24456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.304610 24442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:00:43.304791 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "8570d02d8899451bb4492759213ce62d"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "fd60e7e6a1cbd08129d074438b65e3bb"
server_key_iv: "39e6e030edec97ade3805e211fc2eb34"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.304906 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:43.309727 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:43.310318 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:43.310500 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:43.317345 24442 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:45801
I20260808 06:00:43.317394 24513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:45801 every 8 connection(s)
I20260808 06:00:43.318238 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:43.321403 24514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:43.323330 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24442
I20260808 06:00:43.323458 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:43.323750 14229 external_mini_cluster.cc:1428] Setting key d74acdcc8be1faab03fa5e69a14fc991
I20260808 06:00:43.327358 24514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d: Bootstrap starting.
I20260808 06:00:43.329223 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.325226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.13.229.126:45801)
0808 06:00:43.325857 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.325871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.325905 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:00:43.326629 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.326648 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.326989 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.327331 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.328035 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:43.328039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:43.328051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:43.328060 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:43.328110 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:43.328126 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:43.328288 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:43.328359 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 06:00:43.328585 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260808 06:00:43.330053 24514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:43.330785 24514 log.cc:824] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:43.332588 24514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d: No bootstrap required, opened a new log
I20260808 06:00:43.335846 24514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } }
I20260808 06:00:43.336103 24514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:43.336185 24514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8570d02d8899451bb4492759213ce62d, State: Initialized, Role: FOLLOWER
I20260808 06:00:43.336567 24514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } }
I20260808 06:00:43.336719 24514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:43.336797 24514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:43.336916 24514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:43.337711 24514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } }
I20260808 06:00:43.338035 24514 leader_election.cc:302] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8570d02d8899451bb4492759213ce62d; no voters:
I20260808 06:00:43.338310 24514 leader_election.cc:288] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:43.338441 24519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:43.338673 24519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [term 1 LEADER]: Becoming Leader. State: Replica: 8570d02d8899451bb4492759213ce62d, State: Running, Role: LEADER
I20260808 06:00:43.339004 24519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } }
I20260808 06:00:43.339493 24514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:43.340627 24521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8570d02d8899451bb4492759213ce62d. Latest consensus state: current_term: 1 leader_uuid: "8570d02d8899451bb4492759213ce62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } } }
I20260808 06:00:43.340755 24521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:43.340919 24520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8570d02d8899451bb4492759213ce62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8570d02d8899451bb4492759213ce62d" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45801 } } }
I20260808 06:00:43.341106 24520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:43.341403 24528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:43.343984 24528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:43.349318 24528 catalog_manager.cc:1347] Generated new cluster ID: c4b4f8c1599c4c46bbc06327bcc60fef
I20260808 06:00:43.349377 24528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:43.360880 24528 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:43.361187 24528 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:43.366103 24528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8570d02d8899451bb4492759213ce62d: Generated new TSK 0
I20260808 06:00:43.366819 24528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:00:43.369899 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--builtin_ntp_servers=127.13.229.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 06:00:43.487423 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:43.487645 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:43.487684 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:43.491918 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:43.491966 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:43.492020 24539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:43.496817 24539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:43.497807 24539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:43.499140 24539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:43.508381 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.508379 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.508399 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.508739 24539 server_base.cc:1034] running on GCE node
I20260808 06:00:43.509184 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:43.509756 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:43.510947 24539 hybrid_clock.cc:648] HybridClock initialized: now 1786168843510938 us; error 29 us; skew 500 ppm
I20260808 06:00:43.512738 24539 webserver.cc:460] Webserver started at http://127.13.229.65:40881/ using document root <none> and password file <none>
I20260808 06:00:43.513306 24539 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:43.513399 24539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:43.513629 24539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:43.515585 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "92451d90eb5c416da70d493f75653a51"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "daf6b0f211b60a7e8afac150d7adf9c8"
server_key_iv: "220b9e78aee72e71ef669cca778c7fc1"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.516186 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "92451d90eb5c416da70d493f75653a51"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "daf6b0f211b60a7e8afac150d7adf9c8"
server_key_iv: "220b9e78aee72e71ef669cca778c7fc1"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.519711 24539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:00:43.522172 24553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.523330 24539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:43.523460 24539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "92451d90eb5c416da70d493f75653a51"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "daf6b0f211b60a7e8afac150d7adf9c8"
server_key_iv: "220b9e78aee72e71ef669cca778c7fc1"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.523567 24539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:43.527971 24539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:43.528568 24539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:43.528754 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:43.529290 24539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:43.530161 24539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:43.530232 24539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.530301 24539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:43.530354 24539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.538904 24539 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:41169
I20260808 06:00:43.538928 24672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:41169 every 8 connection(s)
I20260808 06:00:43.539752 24539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:43.545780 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.541659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:60123 (local address 127.13.229.126:45801)
0808 06:00:43.541846 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.541852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.542882 (+ 1030us) server_negotiation.cc:407] Connection header received
0808 06:00:43.544198 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.544201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.544245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.544320 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.545391 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:43.545393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:43.545395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:43.545405 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:43.545422 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:43.545429 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:43.545500 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:43.545639 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 06:00:43.545674 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 06:00:43.546562 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.542122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45801 (local address 127.13.229.65:60123)
0808 06:00:43.542745 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:43.542782 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:43.543999 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:43.544469 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:43.544475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:43.544555 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:43.545162 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:43.545164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:43.545201 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:43.545263 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:43.545505 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:43.545511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:43.545534 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:00:43.545655 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 06:00:43.545826 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260808 06:00:43.546799 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24539
I20260808 06:00:43.546929 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:43.547282 14229 external_mini_cluster.cc:1428] Setting key f0dc9ad83b9c2054a0d0eb7afd87d3e2
I20260808 06:00:43.547920 24673 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45801
I20260808 06:00:43.548141 24673 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:43.548547 24673 heartbeater.cc:511] Master 127.13.229.126:45801 requested a full tablet report, sending...
I20260808 06:00:43.549364 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--builtin_ntp_servers=127.13.229.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 06:00:43.550216 24474 ts_manager.cc:194] Registered new tserver with Master: 92451d90eb5c416da70d493f75653a51 (127.13.229.65:41169)
W20260808 06:00:43.659372 24676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:43.659601 24676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:43.659639 24676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:43.663573 24676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:43.663622 24676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:43.663703 24676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:43.668488 24676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:43.669428 24676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:43.670722 24676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:43.679827 24682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.679881 24676 server_base.cc:1034] running on GCE node
W20260808 06:00:43.679962 24681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.679827 24684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.680419 24676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:43.681021 24676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:43.682204 24676 hybrid_clock.cc:648] HybridClock initialized: now 1786168843682191 us; error 23 us; skew 500 ppm
I20260808 06:00:43.684141 24676 webserver.cc:460] Webserver started at http://127.13.229.66:37471/ using document root <none> and password file <none>
I20260808 06:00:43.684692 24676 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:43.684787 24676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:43.685022 24676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:43.686954 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "6f9b7daca4304c85b7f6f4d8444cf2c7"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "0ca6afe5e6162d34afa876517bb21781"
server_key_iv: "a62b963dd7a1d7d9c1f3612209e5d485"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.687526 24676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "6f9b7daca4304c85b7f6f4d8444cf2c7"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "0ca6afe5e6162d34afa876517bb21781"
server_key_iv: "a62b963dd7a1d7d9c1f3612209e5d485"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.691020 24676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260808 06:00:43.693413 24690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.694612 24676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:00:43.694743 24676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "6f9b7daca4304c85b7f6f4d8444cf2c7"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "0ca6afe5e6162d34afa876517bb21781"
server_key_iv: "a62b963dd7a1d7d9c1f3612209e5d485"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.694851 24676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:43.699232 24676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:43.699801 24676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:43.700022 24676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:43.700543 24676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:43.701406 24676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:43.701474 24676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.701542 24676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:43.701584 24676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.710817 24676 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:38019
I20260808 06:00:43.710856 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:38019 every 8 connection(s)
I20260808 06:00:43.711624 24676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:43.716389 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24676
I20260808 06:00:43.716519 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:43.716809 14229 external_mini_cluster.cc:1428] Setting key 268c85cfcc3c071e85825c7b51983dab
I20260808 06:00:43.717491 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.713351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:45533 (local address 127.13.229.126:45801)
0808 06:00:43.713493 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.713497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.714454 (+ 957us) server_negotiation.cc:407] Connection header received
0808 06:00:43.715703 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.715706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.715745 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.715826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.716954 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:43.716958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:43.716961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:43.716974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:43.717003 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:43.717013 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:43.717117 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:43.717290 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 06:00:43.717356 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:00:43.718400 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.713676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45801 (local address 127.13.229.66:45533)
0808 06:00:43.714331 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:43.714370 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:43.715516 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:43.716012 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:43.716019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:43.716099 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:43.716690 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:43.716691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:43.716728 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:43.716792 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:43.717100 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:43.717109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:43.717115 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:00:43.717287 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 06:00:43.717520 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260808 06:00:43.718827 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--builtin_ntp_servers=127.13.229.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 06:00:43.720082 24809 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45801
I20260808 06:00:43.720305 24809 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:43.720679 24809 heartbeater.cc:511] Master 127.13.229.126:45801 requested a full tablet report, sending...
I20260808 06:00:43.721688 24474 ts_manager.cc:194] Registered new tserver with Master: 6f9b7daca4304c85b7f6f4d8444cf2c7 (127.13.229.66:38019)
W20260808 06:00:43.832145 24812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:43.832391 24812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:43.832466 24812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:43.836547 24812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:43.836596 24812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:43.836699 24812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:43.841723 24812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:43.842715 24812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:43.844061 24812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:43.853396 24818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:43.853411 24817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.853456 24812 server_base.cc:1034] running on GCE node
W20260808 06:00:43.853412 24820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:43.853987 24812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:43.854602 24812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:43.855765 24812 hybrid_clock.cc:648] HybridClock initialized: now 1786168843855747 us; error 27 us; skew 500 ppm
I20260808 06:00:43.857633 24812 webserver.cc:460] Webserver started at http://127.13.229.67:39215/ using document root <none> and password file <none>
I20260808 06:00:43.858196 24812 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:43.858258 24812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:43.858515 24812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:43.860584 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "56144918b4104226856ced159828c54c"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "d2cec1cc6ae5070356c7d4682d109837"
server_key_iv: "271d111ea5e98a5efa89ea69c55527d5"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.861136 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "56144918b4104226856ced159828c54c"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "d2cec1cc6ae5070356c7d4682d109837"
server_key_iv: "271d111ea5e98a5efa89ea69c55527d5"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.864743 24812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:43.867445 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.868662 24812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 06:00:43.868765 24812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "56144918b4104226856ced159828c54c"
format_stamp: "Formatted at 2026-08-08 06:00:43 on dist-test-slave-zkpd"
server_key: "d2cec1cc6ae5070356c7d4682d109837"
server_key_iv: "271d111ea5e98a5efa89ea69c55527d5"
server_key_version: "encryptionkey@0"
I20260808 06:00:43.868850 24812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:43.873224 24812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:43.873792 24812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:43.873924 24812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:43.874387 24812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:43.875214 24812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:43.875262 24812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.875306 24812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:43.875331 24812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:43.883989 24812 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:37867
I20260808 06:00:43.884013 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:37867 every 8 connection(s)
I20260808 06:00:43.884938 24812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:43.886125 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24812
I20260808 06:00:43.886245 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:43.886564 14229 external_mini_cluster.cc:1428] Setting key f8e4ebe640cf2d297cedfe42073ab21d
I20260808 06:00:43.890998 24517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.886816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:51263 (local address 127.13.229.126:45801)
0808 06:00:43.886959 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.886962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.888209 (+ 1247us) server_negotiation.cc:407] Connection header received
0808 06:00:43.889484 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.889487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.889525 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.889592 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.890604 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:43.890607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:43.890608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:00:43.890620 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:43.890639 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:00:43.890647 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:43.890714 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:43.890857 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 06:00:43.890899 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:00:43.891827 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:43.887265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45801 (local address 127.13.229.67:51263)
0808 06:00:43.888057 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:43.888106 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:43.889305 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:43.889753 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:43.889760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:43.889868 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:43.890369 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:00:43.890371 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:43.890408 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:00:43.890473 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:43.890721 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:43.890728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:43.890732 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:00:43.890874 (+ 142us) client_negotiation.cc:240] Negotiation successful
0808 06:00:43.891054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":157,"threads_started":1}
I20260808 06:00:43.892882 24945 heartbeater.cc:347] Connected to a master server at 127.13.229.126:45801
I20260808 06:00:43.893092 24945 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:43.893478 24945 heartbeater.cc:511] Master 127.13.229.126:45801 requested a full tablet report, sending...
I20260808 06:00:43.894507 24474 ts_manager.cc:194] Registered new tserver with Master: 56144918b4104226856ced159828c54c (127.13.229.67:37867)
I20260808 06:00:43.900298 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 06:00:43.903092 24517 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:00:43.902172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.13.229.126:45801)
0808 06:00:43.902303 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:43.902309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:43.902501 (+ 192us) server_negotiation.cc:407] Connection header received
0808 06:00:43.902635 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:43.902638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:43.902676 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:43.902756 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:43.902994 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57588: BlockingRecv error: recv got EOF from 127.0.0.1:57588 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 06:00:43.903314 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24539
I20260808 06:00:43.911621 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24676
I20260808 06:00:43.918447 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24812
I20260808 06:00:43.925050 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24442
2026-08-08T06:00:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (797 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:43 dist-test-slave-zkpd krb5kdc[24955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:43 dist-test-slave-zkpd krb5kdc[24955](info): set up 2 sockets
Aug 08 06:00:43 dist-test-slave-zkpd krb5kdc[24955](info): commencing operation
krb5kdc: starting...
W20260808 06:00:44.979461 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:00:45.105217 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:46205
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:46205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:45.219044 24968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:45.219295 24968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:45.219332 24968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:45.223344 24968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:00:45.223402 24968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:45.223421 24968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:45.223438 24968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:00:45.223495 24968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:00:45.228995 24968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:46205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:46205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:45.230012 24968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:45.231025 24968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:45.239392 24968 server_base.cc:1034] running on GCE node
W20260808 06:00:45.239347 24974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.239322 24973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.239326 24976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:45.240134 24968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:45.241206 24968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:45.242388 24968 hybrid_clock.cc:648] HybridClock initialized: now 1786168845242373 us; error 28 us; skew 500 ppm
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:45.245272 24968 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:00:45.246073 24968 webserver.cc:460] Webserver started at http://127.13.229.126:34119/ using document root <none> and password file <none>
I20260808 06:00:45.246560 24968 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:45.246618 24968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:45.246785 24968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:45.248737 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "b90aadbfdec24f59bfa0cf79370e865c"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "398f8f29043c25eb2a8eb4ca2e65c276"
server_key_iv: "5e34372fd4488ea63ac30e0ac057175f"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.249244 24968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "b90aadbfdec24f59bfa0cf79370e865c"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "398f8f29043c25eb2a8eb4ca2e65c276"
server_key_iv: "5e34372fd4488ea63ac30e0ac057175f"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.252708 24968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:00:45.255064 24983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.256405 24968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:45.256507 24968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "b90aadbfdec24f59bfa0cf79370e865c"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "398f8f29043c25eb2a8eb4ca2e65c276"
server_key_iv: "5e34372fd4488ea63ac30e0ac057175f"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.256584 24968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:45.283057 24968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:45.285782 24968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:45.285940 24968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:45.293064 24968 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:46205
I20260808 06:00:45.293372 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:46205 every 8 connection(s)
I20260808 06:00:45.294059 24968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:00:45.296883 25037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:45.301368 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 24968
I20260808 06:00:45.301501 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:00:45.301807 14229 external_mini_cluster.cc:1428] Setting key 13a5a5032e160fc100a49ee0044fe85c
I20260808 06:00:45.302685 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: Bootstrap starting.
I20260808 06:00:45.305048 25037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:45.305930 25037 log.cc:824] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:45.308151 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: No bootstrap required, opened a new log
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:45.312873 25037 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } }
I20260808 06:00:45.313170 25037 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:45.313234 25037 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90aadbfdec24f59bfa0cf79370e865c, State: Initialized, Role: FOLLOWER
I20260808 06:00:45.313685 25037 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } }
I20260808 06:00:45.313876 25037 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:00:45.313968 25037 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:00:45.314080 25037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:45.315172 25037 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } }
I20260808 06:00:45.315558 25037 leader_election.cc:302] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90aadbfdec24f59bfa0cf79370e865c; no voters:
I20260808 06:00:45.315876 25037 leader_election.cc:288] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:00:45.316027 25042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:45.316277 25042 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [term 1 LEADER]: Becoming Leader. State: Replica: b90aadbfdec24f59bfa0cf79370e865c, State: Running, Role: LEADER
I20260808 06:00:45.316548 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.303151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.13.229.126:46205)
0808 06:00:45.303552 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:45.303568 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:45.303604 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:00:45.304271 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:45.304288 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:45.304608 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:45.304875 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:45.305766 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.306806 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.307977 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.308491 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.308610 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:45.311323 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:45.311331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:45.311353 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:45.311393 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:45.313379 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.313852 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.313857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.313863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.314029 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.314313 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.314319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.314324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.314794 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:45.314979 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:45.315192 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 06:00:45.315550 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260808 06:00:45.316640 25042 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } }
I20260808 06:00:45.317126 25037 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:00:45.319481 25043 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90aadbfdec24f59bfa0cf79370e865c. Latest consensus state: current_term: 1 leader_uuid: "b90aadbfdec24f59bfa0cf79370e865c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } } }
I20260808 06:00:45.319841 25043 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [sys.catalog]: This master's current role is: LEADER
I20260808 06:00:45.319694 25044 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90aadbfdec24f59bfa0cf79370e865c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90aadbfdec24f59bfa0cf79370e865c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46205 } } }
I20260808 06:00:45.320094 25044 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c [sys.catalog]: This master's current role is: LEADER
W20260808 06:00:45.322151 25058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:00:45.322232 25058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:00:45.322300 25059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:00:45.322989 25059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:00:45.328365 25059 catalog_manager.cc:1347] Generated new cluster ID: ce66045567004ebcbf3b39e9e3536227
I20260808 06:00:45.328426 25059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:00:45.341801 25059 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:00:45.342998 25059 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:00:45.347879 25059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: Generated new TSK 0
I20260808 06:00:45.348598 25059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:00:45.386734 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:45.501835 25065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:45.502063 25065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:45.502102 25065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:45.506086 25065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:45.506137 25065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:45.506191 25065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:00:45.511139 25065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:45.512069 25065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:45.513059 25065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:45.522560 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.522572 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:45.522646 25065 server_base.cc:1034] running on GCE node
W20260808 06:00:45.522572 25071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:45.523288 25065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:45.523831 25065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:45.524991 25065 hybrid_clock.cc:648] HybridClock initialized: now 1786168845524977 us; error 24 us; skew 500 ppm
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:45.528607 25065 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:00:45.529773 25065 webserver.cc:460] Webserver started at http://127.13.229.65:34667/ using document root <none> and password file <none>
I20260808 06:00:45.530535 25065 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:45.530618 25065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:45.530870 25065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:45.533061 25065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "b8adbea64b33442b8ca11a408fb6fd76"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "5c4decbf28d3ce2106abe7e17df409b2"
server_key_iv: "151a525f53e85dd4eee8c94b714980ec"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.533618 25065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "b8adbea64b33442b8ca11a408fb6fd76"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "5c4decbf28d3ce2106abe7e17df409b2"
server_key_iv: "151a525f53e85dd4eee8c94b714980ec"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.537160 25065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:00:45.539422 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.540557 25065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:45.540694 25065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "b8adbea64b33442b8ca11a408fb6fd76"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "5c4decbf28d3ce2106abe7e17df409b2"
server_key_iv: "151a525f53e85dd4eee8c94b714980ec"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.540825 25065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:45.557416 25065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:45.560299 25065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:45.560531 25065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:45.561112 25065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:45.561913 25065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:45.561981 25065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.562047 25065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:45.562094 25065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.571241 25065 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:46409
I20260808 06:00:45.571296 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:46409 every 8 connection(s)
I20260808 06:00:45.572156 25065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:00:45.573601 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 25065
I20260808 06:00:45.573750 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:00:45.574083 14229 external_mini_cluster.cc:1428] Setting key 7667c69502f9e40b2c81cdcb57de2398
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:00:45.587039 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.573938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48307 (local address 127.13.229.126:46205)
0808 06:00:45.574073 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:45.574076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:45.575056 (+ 980us) server_negotiation.cc:407] Connection header received
0808 06:00:45.576971 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:45.576974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:45.577017 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:45.577085 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:45.578477 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.579344 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.580550 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.580992 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.581144 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:45.583424 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:45.583430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:45.583436 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:45.583467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:45.585236 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.585699 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.585702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.585704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.585778 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.586076 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.586078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.586079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.586291 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:45.586374 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:45.586802 (+ 428us) server_negotiation.cc:299] Negotiation successful
0808 06:00:45.586917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:00:45.588021 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.574437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:46205 (local address 127.13.229.65:48307)
0808 06:00:45.574904 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:45.574954 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:45.576751 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:45.577299 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:45.577311 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:45.577830 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:45.578332 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:45.578345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.579492 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.579495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:45.580429 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:45.580439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.581133 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.581135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:45.581279 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:45.581860 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:45.581875 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:45.583251 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:45.585351 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:45.585355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:45.585366 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:45.585604 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:45.585872 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:45.585874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:45.585875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:45.585999 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:45.586388 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:45.586393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:45.586605 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 06:00:45.586754 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 06:00:45.586993 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260808 06:00:45.589373 25201 heartbeater.cc:347] Connected to a master server at 127.13.229.126:46205
I20260808 06:00:45.589617 25201 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:45.590154 25201 heartbeater.cc:511] Master 127.13.229.126:46205 requested a full tablet report, sending...
I20260808 06:00:45.591804 24997 ts_manager.cc:194] Registered new tserver with Master: b8adbea64b33442b8ca11a408fb6fd76 (127.13.229.65:46409)
I20260808 06:00:45.593586 24997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:48307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:00:45.604046 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:45.715451 25208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:45.715695 25208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:45.715736 25208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:45.719888 25208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:45.719951 25208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:45.720007 25208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:00:45.724972 25208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:45.725860 25208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:45.726758 25208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:00:45.736250 25208 server_base.cc:1034] running on GCE node
W20260808 06:00:45.736181 25213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.736195 25216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.736186 25214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:45.736982 25208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:45.737622 25208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:45.738781 25208 hybrid_clock.cc:648] HybridClock initialized: now 1786168845738761 us; error 30 us; skew 500 ppm
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:45.741523 25208 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:00:45.742311 25208 webserver.cc:460] Webserver started at http://127.13.229.66:39643/ using document root <none> and password file <none>
I20260808 06:00:45.742823 25208 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:45.742883 25208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:45.743040 25208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:45.745141 25208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "24f5d0326372422f93aaff528faf0fd6"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "bff4ebd0ed706dc72a2d8972d09bdf76"
server_key_iv: "5993cf8e4e36d641e494ce954726d7a2"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.745667 25208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "24f5d0326372422f93aaff528faf0fd6"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "bff4ebd0ed706dc72a2d8972d09bdf76"
server_key_iv: "5993cf8e4e36d641e494ce954726d7a2"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.749346 25208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:00:45.751590 25224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.752795 25208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:00:45.752941 25208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "24f5d0326372422f93aaff528faf0fd6"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "bff4ebd0ed706dc72a2d8972d09bdf76"
server_key_iv: "5993cf8e4e36d641e494ce954726d7a2"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.753050 25208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:45.770155 25208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:45.772990 25208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:45.773197 25208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:45.773810 25208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:45.774655 25208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:45.774726 25208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.774799 25208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:45.774845 25208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.783758 25208 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:34687
I20260808 06:00:45.783785 25340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:34687 every 8 connection(s)
I20260808 06:00:45.784664 25208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:00:45.790458 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 25208
I20260808 06:00:45.790593 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:00:45.790920 14229 external_mini_cluster.cc:1428] Setting key 95dec1fac75a47ed0007a358fab1f55c
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:45.798410 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.786415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:42875 (local address 127.13.229.126:46205)
0808 06:00:45.786552 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:45.786556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:45.787302 (+ 746us) server_negotiation.cc:407] Connection header received
0808 06:00:45.788688 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:45.788691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:45.788736 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:45.788812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:45.789921 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.790448 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.791967 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.792254 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.792354 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:45.794783 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:45.794789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:45.794794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:45.794825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:45.796355 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.796854 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.796857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.796858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.796934 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:45.797293 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:45.797297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:45.797299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:45.797479 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:45.797588 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:45.798122 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 06:00:45.798241 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:00:45.799258 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.786710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:46205 (local address 127.13.229.66:42875)
0808 06:00:45.787145 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:45.787184 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:45.788496 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:45.788985 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:45.788991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:45.789324 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:45.789788 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:45.789800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.790582 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.790586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:45.791807 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:45.791818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:45.792420 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:45.792443 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:45.792596 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:45.793170 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:45.793186 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:45.794612 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:45.796459 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:45.796463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:45.796474 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:45.796718 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:45.797029 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:45.797033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:45.797035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:45.797187 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:45.797587 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:45.797593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:45.797887 (+ 294us) client_negotiation.cc:769] Sending connection context
0808 06:00:45.798076 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 06:00:45.798389 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260808 06:00:45.800397 25341 heartbeater.cc:347] Connected to a master server at 127.13.229.126:46205
I20260808 06:00:45.800652 25341 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:45.801132 25341 heartbeater.cc:511] Master 127.13.229.126:46205 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:00:45.802222 24997 ts_manager.cc:194] Registered new tserver with Master: 24f5d0326372422f93aaff528faf0fd6 (127.13.229.66:34687)
I20260808 06:00:45.802837 24997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:42875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:00:45.819142 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:00:45.929396 25346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:00:45.929646 25346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:00:45.929723 25346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:00:45.933943 25346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:00:45.933998 25346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:00:45.934103 25346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:00:45.939241 25346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:00:45.940287 25346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:00:45.941329 25346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:00:45.950870 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.950827 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:00:45.950873 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:00:45.951210 25346 server_base.cc:1034] running on GCE node
I20260808 06:00:45.951625 25346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:00:45.952245 25346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:00:45.953410 25346 hybrid_clock.cc:648] HybridClock initialized: now 1786168845953399 us; error 27 us; skew 500 ppm
Aug 08 06:00:45 dist-test-slave-zkpd krb5kdc[24955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:00:45.956177 25346 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:00:45.957027 25346 webserver.cc:460] Webserver started at http://127.13.229.67:40589/ using document root <none> and password file <none>
I20260808 06:00:45.957552 25346 fs_manager.cc:359] Metadata directory not provided
I20260808 06:00:45.957636 25346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:00:45.957855 25346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:00:45.959837 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "1761b732c7084da79b7add8e6b3c96d0"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "09e5ca25327469b0b8ade983b3ff7cc2"
server_key_iv: "998460d90b3747a50fbf44167c285896"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.960466 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "1761b732c7084da79b7add8e6b3c96d0"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "09e5ca25327469b0b8ade983b3ff7cc2"
server_key_iv: "998460d90b3747a50fbf44167c285896"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.964115 25346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:00:45.966481 25363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.967598 25346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:00:45.967729 25346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "1761b732c7084da79b7add8e6b3c96d0"
format_stamp: "Formatted at 2026-08-08 06:00:45 on dist-test-slave-zkpd"
server_key: "09e5ca25327469b0b8ade983b3ff7cc2"
server_key_iv: "998460d90b3747a50fbf44167c285896"
server_key_version: "encryptionkey@0"
I20260808 06:00:45.967836 25346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:00:45.979239 25346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:00:45.982228 25346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:00:45.982425 25346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:00:45.983004 25346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:00:45.983821 25346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:00:45.983891 25346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.983981 25346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:00:45.984028 25346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:00:45.993244 25346 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:44059
I20260808 06:00:45.993270 25478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:44059 every 8 connection(s)
I20260808 06:00:45.994171 25346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:00:45.994974 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 25346
I20260808 06:00:45.995106 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:00:45.995438 14229 external_mini_cluster.cc:1428] Setting key 23cfe00f185e439a9287c3a999d556e8
Aug 08 06:00:46 dist-test-slave-zkpd krb5kdc[24955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168845, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:00:46.008193 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.996286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:43387 (local address 127.13.229.126:46205)
0808 06:00:45.996431 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:45.996436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:45.997570 (+ 1134us) server_negotiation.cc:407] Connection header received
0808 06:00:45.998992 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:45.998994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:45.999039 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:45.999104 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:46.000353 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.000879 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.001969 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.002178 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.002255 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.004779 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:46.004782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:46.004784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:46.004807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:46.006302 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:46.006733 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:46.006735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:46.006736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:46.006809 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:46.007129 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:46.007132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:46.007134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:46.007362 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:46.007457 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.007930 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.008050 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:00:46.009158 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:45.996808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:46205 (local address 127.13.229.67:43387)
0808 06:00:45.997401 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:45.997459 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:45.998808 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:45.999290 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:45.999297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:45.999643 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:00:46.000223 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.000238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.000980 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.000983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.001867 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.001876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.002340 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.002344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.002491 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.003039 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:00:46.003055 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:00:46.004625 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:00:46.006399 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:46.006403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:46.006414 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:46.006650 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:46.006887 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:00:46.006889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:00:46.006890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:00:46.007012 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:00:46.007465 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:00:46.007470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:00:46.007684 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.007833 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.008193 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":153,"threads_started":1}
I20260808 06:00:46.010216 25479 heartbeater.cc:347] Connected to a master server at 127.13.229.126:46205
I20260808 06:00:46.010445 25479 heartbeater.cc:464] Registering TS with master...
I20260808 06:00:46.010932 25479 heartbeater.cc:511] Master 127.13.229.126:46205 requested a full tablet report, sending...
I20260808 06:00:46.011962 24997 ts_manager.cc:194] Registered new tserver with Master: 1761b732c7084da79b7add8e6b3c96d0 (127.13.229.67:44059)
I20260808 06:00:46.012503 24997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:43387
I20260808 06:00:46.021116 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:00:46.029917 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.022989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.13.229.126:46205)
0808 06:00:46.023113 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.023116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.023373 (+ 257us) server_negotiation.cc:407] Connection header received
0808 06:00:46.023490 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.023492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.023545 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.023610 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:00:46.024356 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.024841 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.025660 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.025884 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.025965 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.027220 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:00:46.027223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:00:46.027225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:00:46.027247 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:00:46.028527 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:46.028898 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:46.028901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:46.028902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:46.028973 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:00:46.029197 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:00:46.029199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:00:46.029200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:00:46.029426 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:00:46.029535 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.029714 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.029804 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:00:46.032629 24997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:00:46.035763 24997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:00:46.050637 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.044361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:34687 (local address 127.0.0.1:47398)
0808 06:00:46.045344 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.045358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:46.045470 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:46.046252 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:46.046257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:46.046278 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:46.046384 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.046392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047942 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.047946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.049020 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.049030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.050075 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.050191 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050228 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.050338 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.050468 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260808 06:00:46.050769 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.044945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.13.229.66:34687)
0808 06:00:46.045732 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.045746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.045771 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:00:46.045853 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.045858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.046058 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.046212 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:46.046514 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047762 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.049146 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.049932 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050200 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.050519 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.050615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":119,"threads_started":1}
I20260808 06:00:46.050968 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.044257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.13.229.65:46409)
0808 06:00:46.044728 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.044750 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.045635 (+ 885us) server_negotiation.cc:407] Connection header received
0808 06:00:46.045730 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.045734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.045898 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.046063 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:46.046357 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047583 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.049444 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.050052 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050300 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.050712 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.050826 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":80,"threads_started":1}
I20260808 06:00:46.050973 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.045139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.13.229.67:44059)
0808 06:00:46.045965 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.045976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.045995 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:00:46.046063 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.046069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.046288 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.046457 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:46.046655 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047514 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.049423 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.050191 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050292 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050427 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.050702 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.050794 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":147,"threads_started":1}
I20260808 06:00:46.051133 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.044674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:44059 (local address 127.0.0.1:49394)
0808 06:00:46.045624 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.045637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:46.045717 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:46.046441 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:46.046444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:46.046463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:46.046556 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.046563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047640 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.047646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.049289 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.049302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.050304 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.050395 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050419 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.050499 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.050577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":190,"thread_start_us":133,"threads_started":1}
I20260808 06:00:46.051594 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.044105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46409 (local address 127.0.0.1:56240)
0808 06:00:46.045237 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.045286 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:46.045434 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:46.046053 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:46.046059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:46.046081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:46.046226 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.046234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.047702 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.047704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.049296 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.049302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.050362 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.050366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.050448 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.050467 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.050535 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.050592 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":151,"thread_start_us":92,"threads_started":1}
I20260808 06:00:46.054015 25273 tablet_service.cc:1467] Processing CreateTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e (DEFAULT_TABLE table=test-table [id=eeb717d5e40f4c39bd8a731879b87ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:46.054193 25413 tablet_service.cc:1467] Processing CreateTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e (DEFAULT_TABLE table=test-table [id=eeb717d5e40f4c39bd8a731879b87ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:46.054193 25132 tablet_service.cc:1467] Processing CreateTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e (DEFAULT_TABLE table=test-table [id=eeb717d5e40f4c39bd8a731879b87ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:00:46.055130 25273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db7e301b0b4005b5a2d6430d4ccb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:46.055225 25413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db7e301b0b4005b5a2d6430d4ccb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:46.055224 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39db7e301b0b4005b5a2d6430d4ccb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:00:46.062038 25497 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Bootstrap starting.
I20260808 06:00:46.063253 25499 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: Bootstrap starting.
I20260808 06:00:46.064258 25498 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Bootstrap starting.
I20260808 06:00:46.065428 25497 tablet_bootstrap.cc:654] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:46.066568 25497 log.cc:824] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:46.067414 25498 tablet_bootstrap.cc:654] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:46.068883 25498 log.cc:824] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:46.069599 25499 tablet_bootstrap.cc:654] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: Neither blocks nor log segments found. Creating new log.
I20260808 06:00:46.069980 25497 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: No bootstrap required, opened a new log
I20260808 06:00:46.070230 25497 ts_tablet_manager.cc:1397] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 06:00:46.070713 25499 log.cc:824] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: Log is configured to *not* fsync() on all Append() calls
I20260808 06:00:46.072872 25499 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: No bootstrap required, opened a new log
I20260808 06:00:46.073122 25499 ts_tablet_manager.cc:1397] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 06:00:46.075610 25497 raft_consensus.cc:348] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.075979 25497 raft_consensus.cc:374] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:46.076068 25497 raft_consensus.cc:729] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f5d0326372422f93aaff528faf0fd6, State: Initialized, Role: FOLLOWER
I20260808 06:00:46.075979 25499 raft_consensus.cc:348] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.076282 25499 raft_consensus.cc:374] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:46.076390 25499 raft_consensus.cc:729] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8adbea64b33442b8ca11a408fb6fd76, State: Initialized, Role: FOLLOWER
I20260808 06:00:46.076869 25497 consensus_queue.cc:260] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.077666 25341 heartbeater.cc:503] Master 127.13.229.126:46205 was elected leader, sending a full tablet report...
I20260808 06:00:46.077806 25498 tablet_bootstrap.cc:492] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: No bootstrap required, opened a new log
I20260808 06:00:46.078007 25498 ts_tablet_manager.cc:1397] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20260808 06:00:46.079774 25497 ts_tablet_manager.cc:1428] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260808 06:00:46.080525 25499 consensus_queue.cc:260] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.082103 25499 ts_tablet_manager.cc:1428] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260808 06:00:46.082214 25498 raft_consensus.cc:348] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.082549 25201 heartbeater.cc:503] Master 127.13.229.126:46205 was elected leader, sending a full tablet report...
I20260808 06:00:46.082721 25498 raft_consensus.cc:374] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:00:46.082899 25498 raft_consensus.cc:729] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1761b732c7084da79b7add8e6b3c96d0, State: Initialized, Role: FOLLOWER
I20260808 06:00:46.083552 25498 consensus_queue.cc:260] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.084533 25479 heartbeater.cc:503] Master 127.13.229.126:46205 was elected leader, sending a full tablet report...
I20260808 06:00:46.084846 25498 ts_tablet_manager.cc:1428] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 06:00:46.113864 25506 raft_consensus.cc:482] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:00:46.114120 25506 raft_consensus.cc:504] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.115624 25506 leader_election.cc:288] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8adbea64b33442b8ca11a408fb6fd76 (127.13.229.65:46409), 24f5d0326372422f93aaff528faf0fd6 (127.13.229.66:34687)
I20260808 06:00:46.120507 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.115842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:34687 (local address 127.13.229.67:44507)
0808 06:00:46.116051 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.116067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:46.116158 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:46.116546 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:46.116549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:46.116568 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:46.116713 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.116721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.117691 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.117697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.118980 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.118991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.120173 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.120176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.120301 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.120320 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.120381 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.120432 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260808 06:00:46.120522 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.115840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46409 (local address 127.13.229.67:44823)
0808 06:00:46.116569 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.116583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:00:46.116677 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:00:46.116946 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:00:46.116949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:00:46.116968 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:00:46.117072 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.117078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.118174 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.118177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.119315 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:00:46.119328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.120056 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.120059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:00:46.120160 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.120182 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 06:00:46.120291 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:00:46.120365 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":358,"thread_start_us":148,"threads_started":1}
I20260808 06:00:46.120612 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.116040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:44823 (local address 127.13.229.65:46409)
0808 06:00:46.116156 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.116161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.116678 (+ 517us) server_negotiation.cc:407] Connection header received
0808 06:00:46.116766 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.116772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.116832 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.116922 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:46.117181 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.118049 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.119459 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.119889 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.120054 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.120109 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.120344 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.120447 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 06:00:46.121124 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.116016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:44507 (local address 127.13.229.66:34687)
0808 06:00:46.116164 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.116170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.116189 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:00:46.116265 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.116271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.116337 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.116430 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:00:46.116858 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.117555 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.119106 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.119995 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.120139 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.120212 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.120934 (+ 722us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.121014 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:00:46.121285 25154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db7e301b0b4005b5a2d6430d4ccb9e" candidate_uuid: "1761b732c7084da79b7add8e6b3c96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8adbea64b33442b8ca11a408fb6fd76" is_pre_election: true
I20260808 06:00:46.121640 25154 raft_consensus.cc:2454] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1761b732c7084da79b7add8e6b3c96d0 in term 0.
I20260808 06:00:46.121840 25294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db7e301b0b4005b5a2d6430d4ccb9e" candidate_uuid: "1761b732c7084da79b7add8e6b3c96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f5d0326372422f93aaff528faf0fd6" is_pre_election: true
I20260808 06:00:46.122159 25294 raft_consensus.cc:2454] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1761b732c7084da79b7add8e6b3c96d0 in term 0.
I20260808 06:00:46.122198 25365 leader_election.cc:302] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1761b732c7084da79b7add8e6b3c96d0, b8adbea64b33442b8ca11a408fb6fd76; no voters:
I20260808 06:00:46.122532 25506 raft_consensus.cc:2781] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:00:46.122625 25506 raft_consensus.cc:482] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:00:46.122715 25506 raft_consensus.cc:3037] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:46.124389 25506 raft_consensus.cc:504] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.124864 25506 leader_election.cc:288] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [CANDIDATE]: Term 1 election: Requested vote from peers b8adbea64b33442b8ca11a408fb6fd76 (127.13.229.65:46409), 24f5d0326372422f93aaff528faf0fd6 (127.13.229.66:34687)
I20260808 06:00:46.125218 25294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db7e301b0b4005b5a2d6430d4ccb9e" candidate_uuid: "1761b732c7084da79b7add8e6b3c96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24f5d0326372422f93aaff528faf0fd6"
I20260808 06:00:46.125255 25154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39db7e301b0b4005b5a2d6430d4ccb9e" candidate_uuid: "1761b732c7084da79b7add8e6b3c96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8adbea64b33442b8ca11a408fb6fd76"
I20260808 06:00:46.125347 25294 raft_consensus.cc:3037] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:46.125388 25154 raft_consensus.cc:3037] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:00:46.126631 25294 raft_consensus.cc:2454] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1761b732c7084da79b7add8e6b3c96d0 in term 1.
I20260808 06:00:46.126955 25364 leader_election.cc:302] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1761b732c7084da79b7add8e6b3c96d0, 24f5d0326372422f93aaff528faf0fd6; no voters:
I20260808 06:00:46.127007 25154 raft_consensus.cc:2454] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1761b732c7084da79b7add8e6b3c96d0 in term 1.
I20260808 06:00:46.127175 25506 raft_consensus.cc:2781] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:00:46.127444 25506 raft_consensus.cc:686] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 1 LEADER]: Becoming Leader. State: Replica: 1761b732c7084da79b7add8e6b3c96d0, State: Running, Role: LEADER
I20260808 06:00:46.127820 25506 consensus_queue.cc:237] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } }
I20260808 06:00:46.131198 24999 catalog_manager.cc:5495] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1761b732c7084da79b7add8e6b3c96d0 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "1761b732c7084da79b7add8e6b3c96d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1761b732c7084da79b7add8e6b3c96d0" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44059 } health_report { overall_health: HEALTHY } } }
I20260808 06:00:46.154915 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.150686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.13.229.67:44059)
0808 06:00:46.150809 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 06:00:46.150813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:00:46.150826 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:00:46.151004 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:00:46.151006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:00:46.151069 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:00:46.151152 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:00:46.151477 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.151998 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.153403 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:00:46.153597 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:00:46.153687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:00:46.154102 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:00:46.154488 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:00:46.154577 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 06:00:46.154703 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 06:00:46.154751 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:00:46.160915 25154 raft_consensus.cc:1261] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 1 FOLLOWER]: Refusing update from remote peer 1761b732c7084da79b7add8e6b3c96d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:46.160983 25294 raft_consensus.cc:1261] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 1 FOLLOWER]: Refusing update from remote peer 1761b732c7084da79b7add8e6b3c96d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:00:46.161679 25506 consensus_queue.cc:1035] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8adbea64b33442b8ca11a408fb6fd76" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:46.161907 25508 consensus_queue.cc:1035] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f5d0326372422f93aaff528faf0fd6" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 34687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:00:46.172793 25513 mvcc.cc:205] Tried to move back new op lower bound from 7316147593866788864 to 7316147593742426112. Current Snapshot: MvccSnapshot[applied={T|T < 7316147593866788864}]
I20260808 06:00:46.181890 24999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:46.182077 24999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:00:46.185051 24999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90aadbfdec24f59bfa0cf79370e865c: Sending DeleteTablet for 3 replicas of tablet 39db7e301b0b4005b5a2d6430d4ccb9e
I20260808 06:00:46.185858 25132 tablet_service.cc:1514] Processing DeleteTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:46 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:56240
I20260808 06:00:46.186388 25521 tablet_replica.cc:330] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76: stopping tablet replica
I20260808 06:00:46.186388 25413 tablet_service.cc:1514] Processing DeleteTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:46 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:49394
I20260808 06:00:46.186815 25521 raft_consensus.cc:2229] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:46.187191 25521 raft_consensus.cc:2258] T 39db7e301b0b4005b5a2d6430d4ccb9e P b8adbea64b33442b8ca11a408fb6fd76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:46.187304 25273 tablet_service.cc:1514] Processing DeleteTablet for tablet 39db7e301b0b4005b5a2d6430d4ccb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:00:46 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:47398
I20260808 06:00:46.188283 25523 tablet_replica.cc:330] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: stopping tablet replica
I20260808 06:00:46.188444 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 25065
I20260808 06:00:46.188602 25523 raft_consensus.cc:2229] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:00:46.190388 25523 raft_consensus.cc:2258] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:46.191030 25522 tablet_replica.cc:330] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: stopping tablet replica
I20260808 06:00:46.191264 25522 raft_consensus.cc:2229] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:00:46.191625 25522 raft_consensus.cc:2258] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:00:46.193065 25523 ts_tablet_manager.cc:1905] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:46.195772 25523 ts_tablet_manager.cc:1918] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:46.195858 25523 log.cc:1196] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/39db7e301b0b4005b5a2d6430d4ccb9e
I20260808 06:00:46.196203 25523 ts_tablet_manager.cc:1939] T 39db7e301b0b4005b5a2d6430d4ccb9e P 24f5d0326372422f93aaff528faf0fd6: Deleting consensus metadata
I20260808 06:00:46.197472 24984 catalog_manager.cc:4841] TS 24f5d0326372422f93aaff528faf0fd6 (127.13.229.66:34687): tablet 39db7e301b0b4005b5a2d6430d4ccb9e (table test-table [id=eeb717d5e40f4c39bd8a731879b87ff9]) successfully deleted
I20260808 06:00:46.201154 25522 ts_tablet_manager.cc:1905] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:00:46.203807 25522 ts_tablet_manager.cc:1918] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:00:46.203995 25522 log.cc:1196] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/39db7e301b0b4005b5a2d6430d4ccb9e
I20260808 06:00:46.204387 25522 ts_tablet_manager.cc:1939] T 39db7e301b0b4005b5a2d6430d4ccb9e P 1761b732c7084da79b7add8e6b3c96d0: Deleting consensus metadata
I20260808 06:00:46.205564 24986 catalog_manager.cc:4841] TS 1761b732c7084da79b7add8e6b3c96d0 (127.13.229.67:44059): tablet 39db7e301b0b4005b5a2d6430d4ccb9e (table test-table [id=eeb717d5e40f4c39bd8a731879b87ff9]) successfully deleted
W20260808 06:00:46.207027 24985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:46409 (error 108)
I20260808 06:00:46.207940 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 25208
I20260808 06:00:46.217425 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:00:46.217045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46409 (local address 127.0.0.1:56242)
0808 06:00:46.217213 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 06:00:46.217312 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:00:46.217645 24985 catalog_manager.cc:4584] TS b8adbea64b33442b8ca11a408fb6fd76 (127.13.229.65:46409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39db7e301b0b4005b5a2d6430d4ccb9e: Network error: Client connection negotiation failed: client connection to 127.13.229.65:46409: connect: Connection refused (error 111)
I20260808 06:00:46.219136 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 25346
I20260808 06:00:46.227758 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 24968
2026-08-08T06:00:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2303 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:00:46 dist-test-slave-zkpd krb5kdc[25527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:00:46 dist-test-slave-zkpd krb5kdc[25527](info): set up 2 sockets
Aug 08 06:00:46 dist-test-slave-zkpd krb5kdc[25527](info): commencing operation
krb5kdc: starting...
W20260808 06:00:47.302665 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:00:47 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:00:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangeradmin_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangeradmin_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangerlookup_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangerlookup_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
I20260808 06:00:47.477207 14229 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNODW_f/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/postgres -l logfile start
2026-08-08 06:00:49.854 UTC [25560] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 06:00:49.854 UTC [25560] LOG: listening on IPv4 address "127.13.229.84", port 38719
2026-08-08 06:00:49.857 UTC [25560] LOG: listening on Unix socket "/tmp/.s.PGSQL.38719"
2026-08-08 06:00:49.879 UTC [25563] LOG: database system was shut down at 2026-08-08 06:00:48 UTC
2026-08-08 06:00:49.882 UTC [25560] LOG: database system is ready to accept connections
I20260808 06:00:50.865969 14229 mini_postgres.cc:96] Postgres bound to 38719
2026-08-08 06:00:50.873 UTC [25571] FATAL: database "slave" does not exist
127.13.229.84:38719 - accepting connections
I20260808 06:00:50.874470 14229 mini_ranger.cc:161] Starting Ranger...
I20260808 06:00:50.890158 14229 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 06:00:51.345112 14229 mini_ranger.cc:87] Created ranger Postgres database
I20260808 06:00:51.345225 14229 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin
2026-08-08 06:00:51,620 [I] DB FLAVOR :POSTGRES
2026-08-08 06:00:51,620 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:00:51,620 [I] Checking connection..
2026-08-08 06:00:51,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 06:00:51,881 [I] Checking connection passed.
2026-08-08 06:00:51,881 [I] --------- Verifying version history table ---------
2026-08-08 06:00:51,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:00:52,128 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:00:52,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:00:52,376 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:00:52,376 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 06:00:52,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 06:00:52.625 UTC [25640] WARNING: there is no transaction in progress
2026-08-08 06:00:52,632 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 06:00:52,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:00:52,878 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 06:00:52,878 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:00:52,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 06:00:53,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 06:00:53,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zkpd.c.gcp-upstream.internal','N') ;"
2026-08-08 06:00:53,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:00:53,863 [I] Table x_portal_user does not exist in database ranger
2026-08-08 06:00:53,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:00:54,091 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 06:00:54,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:00:54,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:00:54,546 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 06:00:54,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 06:00:55.440 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.456 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.473 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.488 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.505 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.525 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.589 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.595 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.609 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.618 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.628 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.642 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.651 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.657 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.663 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:55.669 UTC [25753] WARNING: there is no transaction in progress
2026-08-08 06:00:56,258 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 06:00:56,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 06:00:56,505 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 06:00:56,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:00:56,743 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 06:00:56,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:00:57,001 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 06:00:57,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:00:57,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:00:57,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zkpd.c.gcp-upstream.internal';"
2026-08-08 06:00:57,715 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 06:00:57,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 06:00:57,951 [I] DB_PATCHES have already been applied
I20260808 06:00:57.956956 14229 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNODW_f/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 06:00:57.957018 14229 mini_ranger.cc:193] Using host: 127.13.229.84
I20260808 06:00:57.961352 14229 mini_ranger.cc:227] Ranger admin URL: http://127.13.229.84:41371
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:00:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 06:00:59.101306 14229 mini_ranger.cc:160] Time spent starting Ranger: real 8.227s user 0.002s sys 0.010s
Aug 08, 2026 6:01:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:01:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:01:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:01:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:01:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 06:01:19.847568 14229 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangerkms_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/rangerkms_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.229.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.229.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 06:01:19 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168879, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 06:01:19.937567 14229 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 06:01:19.947446 14229 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 06:01:20.762275 14229 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 06:01:20.762387 14229 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms
I20260808 06:01:20.762501 14229 mini_ranger_kms.cc:227] Using postgres at 127.13.229.84:38719
2026-08-08 06:01:20,980 [I] DB FLAVOR :POSTGRES
2026-08-08 06:01:20,980 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:01:20,980 [I] Checking connection
2026-08-08 06:01:20,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 06:01:21,224 [I] connection success
2026-08-08 06:01:21,224 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 06:01:21,224 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 06:01:21,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 06:01:21,475 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 06:01:21,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:01:21,475 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 06:01:21,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:38719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 06:01:21,757 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 06:01:22.216655 14229 mini_ranger_kms.cc:310] Created kms service
I20260808 06:01:22.450605 14229 mini_ranger_kms.cc:326] Created kudu user
I20260808 06:01:22.497191 14229 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 06:01:22 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168879, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:22.889814 14229 mini_ranger_kms.cc:384] Added ranger policy
I20260808 06:01:22.889986 14229 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 06:01:22.890013 14229 mini_ranger_kms.cc:240] Using host: 127.13.229.84
I20260808 06:01:22.895730 14229 mini_ranger_kms.cc:276] Ranger KMS PID: 25946
I20260808 06:01:22.895871 14229 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.229.84:56829
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:01:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 06:01:25.226794 14229 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.289s user 0.008s sys 0.195s
I20260808 06:01:25.226923 14229 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 06:01:25.227025 14229 mini_ranger_kms.cc:401] 127.13.229.84:56829/kms/v1/keys
Aug 08 06:01:25 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786168885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 06:01:26 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
Aug 08 06:01:26 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
Aug 08 06:01:28 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:01:28.742993 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36295
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:36295
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56829/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:01:28.882874 26000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:01:28.883167 26000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:01:28.883208 26000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:01:28.887348 26000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:01:28.887411 26000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:01:28.887430 26000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:01:28.887447 26000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:01:28.887488 26000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:01:28.893093 26000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.229.84:56829/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:36295
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:01:28.894217 26000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:01:28.895298 26000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:01:28.905369 26008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:01:28.905472 26009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:28.905616 26000 server_base.cc:1034] running on GCE node
W20260808 06:01:28.905375 26011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:28.906379 26000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:01:28.907725 26000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:01:28.908921 26000 hybrid_clock.cc:648] HybridClock initialized: now 1786168888908893 us; error 40 us; skew 500 ppm
Aug 08 06:01:28 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168888, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:01:28.916471 26000 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:01:28.917469 26000 webserver.cc:460] Webserver started at http://127.13.229.126:46553/ using document root <none> and password file <none>
I20260808 06:01:28.918044 26000 fs_manager.cc:359] Metadata directory not provided
I20260808 06:01:28.918100 26000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:01:28.918316 26000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:01:28 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168888, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:28.979218 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "7f3b6d143f324e699f3c2fa62bdb9c68"
format_stamp: "Formatted at 2026-08-08 06:01:28 on dist-test-slave-zkpd"
server_key: "8c2b598b9936bf4a387ba299b4715898"
server_key_iv: "9b1baca78daed8b23430ae710842bb0c"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:28.979851 26000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "7f3b6d143f324e699f3c2fa62bdb9c68"
format_stamp: "Formatted at 2026-08-08 06:01:28 on dist-test-slave-zkpd"
server_key: "8c2b598b9936bf4a387ba299b4715898"
server_key_iv: "9b1baca78daed8b23430ae710842bb0c"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:28.983834 26000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 06:01:28.998034 26018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:28.999488 26000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:01:28.999626 26000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "7f3b6d143f324e699f3c2fa62bdb9c68"
format_stamp: "Formatted at 2026-08-08 06:01:28 on dist-test-slave-zkpd"
server_key: "8c2b598b9936bf4a387ba299b4715898"
server_key_iv: "9b1baca78daed8b23430ae710842bb0c"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:28.999778 26000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:01:29.027781 26000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:01:29.031255 26000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:01:29.031518 26000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:01:29.041050 26000 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:36295
I20260808 06:01:29.041074 26073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:36295 every 8 connection(s)
I20260808 06:01:29.042063 26000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:01:29.044818 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:01:29.050344 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: Bootstrap starting.
I20260808 06:01:29.051399 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 26000
I20260808 06:01:29.051542 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:01:29.051826 14229 external_mini_cluster.cc:1428] Setting key a60173a1b31c9560125188b39e5b72b2
I20260808 06:01:29.053143 26074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: Neither blocks nor log segments found. Creating new log.
I20260808 06:01:29.054083 26074 log.cc:824] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: Log is configured to *not* fsync() on all Append() calls
I20260808 06:01:29.056298 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: No bootstrap required, opened a new log
I20260808 06:01:29.060808 26074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } }
I20260808 06:01:29.061103 26074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:01:29.061163 26074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3b6d143f324e699f3c2fa62bdb9c68, State: Initialized, Role: FOLLOWER
Aug 08 06:01:29 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:01:29.061615 26074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } }
I20260808 06:01:29.061769 26074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:01:29.061887 26074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:01:29.062000 26074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:01:29.063297 26074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } }
I20260808 06:01:29.063673 26074 leader_election.cc:302] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3b6d143f324e699f3c2fa62bdb9c68; no voters:
I20260808 06:01:29.064088 26074 leader_election.cc:288] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:01:29.064301 26079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:01:29.064983 26079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3b6d143f324e699f3c2fa62bdb9c68, State: Running, Role: LEADER
I20260808 06:01:29.065472 26074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:01:29.065464 26079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } }
I20260808 06:01:29.067406 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:29.053005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.13.229.126:36295)
0808 06:01:29.053506 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:29.053525 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:29.053568 (+ 43us) server_negotiation.cc:407] Connection header received
0808 06:01:29.054461 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:29.054490 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:29.054872 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:29.055246 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:01:29.055933 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:29.057397 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:29.058241 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:29.058988 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:29.059151 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:29.061895 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:01:29.061905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:01:29.061924 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:01:29.061960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:01:29.064042 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:29.064608 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:29.064614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:29.064620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:29.064731 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:29.065044 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:29.065047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:29.065049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:29.065634 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:01:29.065842 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:29.066131 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 06:01:29.066398 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":173,"threads_started":1}
I20260808 06:01:29.068064 26074 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 06:01:29.068126 26080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } } }
I20260808 06:01:29.068234 26080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [sys.catalog]: This master's current role is: LEADER
I20260808 06:01:29.068770 26081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3b6d143f324e699f3c2fa62bdb9c68. Latest consensus state: current_term: 1 leader_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3b6d143f324e699f3c2fa62bdb9c68" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36295 } } }
I20260808 06:01:29.068930 26081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68 [sys.catalog]: This master's current role is: LEADER
I20260808 06:01:29.072058 26095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:01:29.076056 26095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:01:29.081954 26095 catalog_manager.cc:1347] Generated new cluster ID: a23cc6f0471c4c998325accf0ab12f5e
I20260808 06:01:29.082038 26095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:01:29.102089 26095 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:01:29.103415 26095 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:01:29.113463 26095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: Generated new TSK 0
I20260808 06:01:29.114286 26095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:01:29.829303 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:01:30.000517 26122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:01:30.000792 26122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:01:30.000864 26122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:01:30.005458 26122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:01:30.005597 26122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:01:30.005717 26122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:01:30.013717 26122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.229.84:56829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:01:30.015077 26122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:01:30.016304 26122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:01:30.026139 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:30.026531 26122 server_base.cc:1034] running on GCE node
W20260808 06:01:30.026731 26134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:01:30.026361 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:30.027281 26122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:01:30.028071 26122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:01:30.029244 26122 hybrid_clock.cc:648] HybridClock initialized: now 1786168890029210 us; error 54 us; skew 500 ppm
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:01:30.032150 26122 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:01:30.033159 26122 webserver.cc:460] Webserver started at http://127.13.229.65:33655/ using document root <none> and password file <none>
I20260808 06:01:30.033754 26122 fs_manager.cc:359] Metadata directory not provided
I20260808 06:01:30.033845 26122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:01:30.034070 26122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:30.059186 26122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "a2971c2ede204cee8b7d85e263e1c347"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "60bff0d27ef0fd9e42a8212ee829322d"
server_key_iv: "44cb04de64c54237779e6558d41b8656"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.059989 26122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "a2971c2ede204cee8b7d85e263e1c347"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "60bff0d27ef0fd9e42a8212ee829322d"
server_key_iv: "44cb04de64c54237779e6558d41b8656"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.065256 26122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 06:01:30.090495 26143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.092365 26122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 06:01:30.092507 26122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "a2971c2ede204cee8b7d85e263e1c347"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "60bff0d27ef0fd9e42a8212ee829322d"
server_key_iv: "44cb04de64c54237779e6558d41b8656"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.092655 26122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:01:30.128095 26122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:01:30.132099 26122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:01:30.132365 26122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:01:30.133165 26122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:01:30.134325 26122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:01:30.134436 26122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.134562 26122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:01:30.134642 26122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.146035 26122 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:46353
I20260808 06:01:30.146711 26264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:46353 every 8 connection(s)
I20260808 06:01:30.147310 26122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:01:30.155786 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 26122
I20260808 06:01:30.155951 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:01:30.156268 14229 external_mini_cluster.cc:1428] Setting key 4a95daf854dad7b468820b04c2031807
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:01:30.163968 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.150297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36295 (local address 127.13.229.65:46047)
0808 06:01:30.150813 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.150867 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.151783 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.152481 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.152488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.152797 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:01:30.153313 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.153328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.154201 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.154204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.155084 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.155093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.155602 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.155607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.155785 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.156453 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:01:30.156469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:01:30.158111 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:01:30.160547 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.160551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.160563 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.160812 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.161177 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.161179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.161181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.161321 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.161864 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:01:30.161874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:01:30.162208 (+ 334us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.162423 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.162807 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260808 06:01:30.165499 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.149813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:46047 (local address 127.13.229.126:36295)
0808 06:01:30.150080 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.150084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.150977 (+ 893us) server_negotiation.cc:407] Connection header received
0808 06:01:30.152049 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.152052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.152115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.152239 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:01:30.153467 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.154068 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.155213 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.155450 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.155562 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.158325 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:01:30.158329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:01:30.158331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:01:30.158365 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:01:30.160427 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.160959 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.160963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.160965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.161073 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.161418 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.161423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.161425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.161691 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:01:30.161805 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.165141 (+ 3336us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.165294 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260808 06:01:30.166827 26265 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36295
I20260808 06:01:30.167073 26265 heartbeater.cc:464] Registering TS with master...
I20260808 06:01:30.167627 26265 heartbeater.cc:511] Master 127.13.229.126:36295 requested a full tablet report, sending...
I20260808 06:01:30.169485 26037 ts_manager.cc:194] Registered new tserver with Master: a2971c2ede204cee8b7d85e263e1c347 (127.13.229.65:46353)
I20260808 06:01:30.171166 26037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:46047
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:01:30.190923 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 06:01:30.349889 26271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:01:30.350270 26271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:01:30.350359 26271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:01:30.356647 26271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:01:30.356721 26271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:01:30.356827 26271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:01:30.364814 26271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.229.84:56829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:01:30.366242 26271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:01:30.367710 26271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:01:30.380784 26278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:01:30.381210 26277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:30.381440 26271 server_base.cc:1034] running on GCE node
W20260808 06:01:30.381946 26281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:30.382481 26271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:01:30.383301 26271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:01:30.386597 26271 hybrid_clock.cc:648] HybridClock initialized: now 1786168890386511 us; error 91 us; skew 500 ppm
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:01:30.390645 26271 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:01:30.391827 26271 webserver.cc:460] Webserver started at http://127.13.229.66:34615/ using document root <none> and password file <none>
I20260808 06:01:30.392621 26271 fs_manager.cc:359] Metadata directory not provided
I20260808 06:01:30.392777 26271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:01:30.393095 26271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:30.409480 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "a518ef4d900505082c9d8fb289208f85"
server_key_iv: "0fb8187f0fce41c14aca3f117f73ad1a"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.410287 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "a518ef4d900505082c9d8fb289208f85"
server_key_iv: "0fb8187f0fce41c14aca3f117f73ad1a"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.417565 26271 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 06:01:30.436282 26290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.437945 26271 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 06:01:30.438081 26271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "a518ef4d900505082c9d8fb289208f85"
server_key_iv: "0fb8187f0fce41c14aca3f117f73ad1a"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.438202 26271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:01:30.454345 26271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:01:30.458248 26271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:01:30.458535 26271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:01:30.459450 26271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:01:30.460595 26271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:01:30.460704 26271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.460819 26271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:01:30.460886 26271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.476455 26271 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:33243
I20260808 06:01:30.477751 26271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:01:30.479466 26407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:33243 every 8 connection(s)
I20260808 06:01:30.483940 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 26271
I20260808 06:01:30.484083 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:01:30.484400 14229 external_mini_cluster.cc:1428] Setting key 8f32c567ba2f2f2206b7a598a30aa5af
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:01:30.500269 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.483105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:38127 (local address 127.13.229.126:36295)
0808 06:01:30.483274 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.483279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.484145 (+ 866us) server_negotiation.cc:407] Connection header received
0808 06:01:30.485721 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.485724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.485796 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.485851 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:01:30.487515 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.488418 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.490625 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.491016 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.491091 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.494993 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:01:30.495000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:01:30.495006 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:01:30.495041 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:01:30.497095 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.497960 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.497967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.497970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.498075 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.498567 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.498573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.498577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.498904 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:01:30.498990 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.499914 (+ 924us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.500067 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:01:30.502301 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.481912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36295 (local address 127.13.229.66:38127)
0808 06:01:30.483873 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.483964 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.485466 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.486283 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.486292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.486645 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:01:30.487338 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.487359 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.488582 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.488586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.490481 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.490493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.491349 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.491353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.491549 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.492868 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:01:30.492893 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:01:30.494787 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:01:30.497250 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.497260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.497281 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.497823 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.498210 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.498216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.498220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.498434 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.499167 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:01:30.499177 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:01:30.499548 (+ 371us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.500769 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.501190 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:01:30.504482 26408 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36295
I20260808 06:01:30.504765 26408 heartbeater.cc:464] Registering TS with master...
I20260808 06:01:30.505281 26408 heartbeater.cc:511] Master 127.13.229.126:36295 requested a full tablet report, sending...
I20260808 06:01:30.506399 26037 ts_manager.cc:194] Registered new tserver with Master: 4a6321ef11ad4dc0a9d0c52b9284f42a (127.13.229.66:33243)
I20260808 06:01:30.506994 26037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:38127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:01:30.526808 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56829/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:01:30.697190 26413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:01:30.697608 26413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:01:30.697747 26413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:01:30.704727 26413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:01:30.704905 26413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:01:30.705053 26413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:30.712356 26413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.229.84:56829/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:01:30.713889 26413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:01:30.715291 26413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:01:30.735787 26419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:01:30.735801 26420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:01:30.737630 26422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:01:30.737897 26413 server_base.cc:1034] running on GCE node
I20260808 06:01:30.738430 26413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:01:30.739262 26413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:01:30.742522 26413 hybrid_clock.cc:648] HybridClock initialized: now 1786168890742431 us; error 95 us; skew 500 ppm
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:01:30.746546 26413 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:01:30.747680 26413 webserver.cc:460] Webserver started at http://127.13.229.67:44261/ using document root <none> and password file <none>
I20260808 06:01:30.748466 26413 fs_manager.cc:359] Metadata directory not provided
I20260808 06:01:30.748608 26413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:01:30.748899 26413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:01:30.773383 26413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "d3c738e04044481499ffd42d5c05e7be"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "bae6dbfdda655a26a0a506a2e96656a3"
server_key_iv: "3f21092eba6d4007e41ac5d17d848417"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.774250 26413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "d3c738e04044481499ffd42d5c05e7be"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "bae6dbfdda655a26a0a506a2e96656a3"
server_key_iv: "3f21092eba6d4007e41ac5d17d848417"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.780781 26413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 06:01:30.800335 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.802160 26413 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 06:01:30.802312 26413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "d3c738e04044481499ffd42d5c05e7be"
format_stamp: "Formatted at 2026-08-08 06:01:30 on dist-test-slave-zkpd"
server_key: "bae6dbfdda655a26a0a506a2e96656a3"
server_key_iv: "3f21092eba6d4007e41ac5d17d848417"
server_key_version: "kuduclusterkey@0"
I20260808 06:01:30.802438 26413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:01:30.822297 26413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:01:30.825678 26413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:01:30.825906 26413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:01:30.826579 26413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:01:30.827551 26413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:01:30.827623 26413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.827728 26413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:01:30.827785 26413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:01:30.841501 26413 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:38703
I20260808 06:01:30.842543 26413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:01:30.849017 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 26413
I20260808 06:01:30.849073 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:38703 every 8 connection(s)
I20260808 06:01:30.849166 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:01:30.849432 14229 external_mini_cluster.cc:1428] Setting key 90ccf1d7f04f700c8a8f2c88c34c7c89
Aug 08 06:01:30 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:01:30.865895 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.853091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:42839 (local address 127.13.229.126:36295)
0808 06:01:30.853249 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.853255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.853273 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:01:30.853942 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.853946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.854000 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.854052 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:01:30.855774 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.856770 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.858470 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.858797 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.858907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.861911 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:01:30.861916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:01:30.861920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:01:30.861952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:01:30.863507 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.864092 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.864096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.864098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.864175 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.864564 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.864567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.864569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.864798 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:01:30.864884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.865585 (+ 701us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.865705 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 06:01:30.867429 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.851835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36295 (local address 127.13.229.67:42839)
0808 06:01:30.852549 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.852608 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.853676 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.854486 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.854496 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.854854 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:01:30.855583 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.855602 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.856937 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.856940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.858320 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.858332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.859476 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.859486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.859651 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.860314 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:01:30.860330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:01:30.861731 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:01:30.863643 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.863649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.863665 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.863970 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.864267 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:01:30.864271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:01:30.864273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:01:30.864437 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:01:30.864954 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:01:30.864963 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:01:30.865298 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.866128 (+ 830us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.866481 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":200,"threads_started":1}
I20260808 06:01:30.868726 26546 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36295
I20260808 06:01:30.868990 26546 heartbeater.cc:464] Registering TS with master...
I20260808 06:01:30.869508 26546 heartbeater.cc:511] Master 127.13.229.126:36295 requested a full tablet report, sending...
I20260808 06:01:30.870666 26037 ts_manager.cc:194] Registered new tserver with Master: d3c738e04044481499ffd42d5c05e7be (127.13.229.67:38703)
I20260808 06:01:30.871241 26037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:42839
I20260808 06:01:30.877997 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:01:30.887846 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.880224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.13.229.126:36295)
0808 06:01:30.880353 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.880360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.880641 (+ 281us) server_negotiation.cc:407] Connection header received
0808 06:01:30.880773 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.880776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.880820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.880917 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:01:30.881617 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.882130 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.882967 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.883193 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.883304 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.884702 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:01:30.884705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:01:30.884708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:01:30.884734 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:01:30.886216 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.886650 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.886654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.886656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.886736 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:01:30.887010 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:01:30.887014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:01:30.887016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:01:30.887188 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:01:30.887333 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.887563 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.887681 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:01:30.890763 26037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:01:30.894034 26037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:01:30.909055 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.903072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46353 (local address 127.0.0.1:33196)
0808 06:01:30.904193 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.904243 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.904375 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.904967 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.904974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.905004 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:01:30.905190 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.905198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.906635 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.906640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.907496 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.907506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.908526 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.908531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.908646 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.908700 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.908805 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.908943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":108,"threads_started":1}
I20260808 06:01:30.910992 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.904018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.13.229.65:46353)
0808 06:01:30.904457 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.904467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.904490 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:01:30.904563 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.904567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.904788 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.905365 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:01:30.905522 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.906516 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.907621 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.908340 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.909386 (+ 1046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.910708 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.910760 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.910853 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":119,"threads_started":1}
I20260808 06:01:30.914217 26189 tablet_service.cc:1467] Processing CreateTablet for tablet b457587b3efd47f58ce19171dd4a6258 (DEFAULT_TABLE table=test-table [id=d81c244a5c4b41ff973b19d0b0b508bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:01:30.915439 26189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b457587b3efd47f58ce19171dd4a6258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:01:30.916404 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.903222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33243 (local address 127.0.0.1:41410)
0808 06:01:30.909427 (+ 6205us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.909442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.909553 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.910210 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.910216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.910239 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:01:30.910334 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.910340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.912813 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.912817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.913935 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.913947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.915181 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.915185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.916099 (+ 914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.916127 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.916202 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.916275 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6123,"thread_start_us":105,"threads_started":1}
I20260808 06:01:30.916935 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.904514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.13.229.66:33243)
0808 06:01:30.904973 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.904986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.909553 (+ 4567us) server_negotiation.cc:407] Connection header received
0808 06:01:30.909673 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.909679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.909913 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.910064 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:01:30.910469 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.912652 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.914104 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.914874 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.914978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.915116 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.916690 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.916801 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260808 06:01:30.921082 26340 tablet_service.cc:1467] Processing CreateTablet for tablet b457587b3efd47f58ce19171dd4a6258 (DEFAULT_TABLE table=test-table [id=d81c244a5c4b41ff973b19d0b0b508bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:01:30.922227 26340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b457587b3efd47f58ce19171dd4a6258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:01:30.928426 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.903847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38703 (local address 127.0.0.1:58946)
0808 06:01:30.909700 (+ 5853us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:30.909713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:30.909788 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:30.910469 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:30.910472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:30.910488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:01:30.910566 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.910571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.925375 (+ 14804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.925379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.926459 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:30.926472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.927987 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.927991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:30.928076 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.928103 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:01:30.928179 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 06:01:30.928257 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5796,"thread_start_us":104,"threads_started":1}
I20260808 06:01:30.929138 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:30.904913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.13.229.67:38703)
0808 06:01:30.905258 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:30.905273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:30.909880 (+ 4607us) server_negotiation.cc:407] Connection header received
0808 06:01:30.909967 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:30.909977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:30.910162 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:30.910319 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:01:30.923991 (+ 13672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.925191 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.926780 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:30.927609 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:30.927735 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:30.927919 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:30.928892 (+ 973us) server_negotiation.cc:299] Negotiation successful
0808 06:01:30.928998 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260808 06:01:30.933363 26565 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Bootstrap starting.
I20260808 06:01:30.932937 26462 tablet_service.cc:1467] Processing CreateTablet for tablet b457587b3efd47f58ce19171dd4a6258 (DEFAULT_TABLE table=test-table [id=d81c244a5c4b41ff973b19d0b0b508bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:01:30.934060 26462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b457587b3efd47f58ce19171dd4a6258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:01:30.936034 26565 tablet_bootstrap.cc:654] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Neither blocks nor log segments found. Creating new log.
I20260808 06:01:30.937198 26565 log.cc:824] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Log is configured to *not* fsync() on all Append() calls
I20260808 06:01:30.937645 26566 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Bootstrap starting.
I20260808 06:01:30.941733 26566 tablet_bootstrap.cc:654] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Neither blocks nor log segments found. Creating new log.
I20260808 06:01:30.942886 26566 log.cc:824] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Log is configured to *not* fsync() on all Append() calls
I20260808 06:01:30.942970 26568 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Bootstrap starting.
I20260808 06:01:30.945235 26565 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: No bootstrap required, opened a new log
I20260808 06:01:30.945482 26565 ts_tablet_manager.cc:1397] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260808 06:01:30.950031 26565 raft_consensus.cc:348] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.950469 26565 raft_consensus.cc:374] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:01:30.950575 26565 raft_consensus.cc:729] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2971c2ede204cee8b7d85e263e1c347, State: Initialized, Role: FOLLOWER
I20260808 06:01:30.951052 26568 tablet_bootstrap.cc:654] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Neither blocks nor log segments found. Creating new log.
I20260808 06:01:30.952157 26568 log.cc:824] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Log is configured to *not* fsync() on all Append() calls
I20260808 06:01:30.952147 26565 consensus_queue.cc:260] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.953222 26565 ts_tablet_manager.cc:1428] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260808 06:01:30.953629 26265 heartbeater.cc:503] Master 127.13.229.126:36295 was elected leader, sending a full tablet report...
I20260808 06:01:30.955415 26566 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: No bootstrap required, opened a new log
I20260808 06:01:30.955611 26566 ts_tablet_manager.cc:1397] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20260808 06:01:30.959378 26566 raft_consensus.cc:348] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.959767 26566 raft_consensus.cc:374] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:01:30.959849 26566 raft_consensus.cc:729] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6321ef11ad4dc0a9d0c52b9284f42a, State: Initialized, Role: FOLLOWER
I20260808 06:01:30.960388 26566 consensus_queue.cc:260] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.961354 26566 ts_tablet_manager.cc:1428] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260808 06:01:30.961726 26408 heartbeater.cc:503] Master 127.13.229.126:36295 was elected leader, sending a full tablet report...
I20260808 06:01:30.966559 26568 tablet_bootstrap.cc:492] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: No bootstrap required, opened a new log
I20260808 06:01:30.966780 26568 ts_tablet_manager.cc:1397] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20260808 06:01:30.969961 26568 raft_consensus.cc:348] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.970196 26568 raft_consensus.cc:374] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:01:30.970268 26568 raft_consensus.cc:729] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c738e04044481499ffd42d5c05e7be, State: Initialized, Role: FOLLOWER
I20260808 06:01:30.970723 26568 consensus_queue.cc:260] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:30.971347 26546 heartbeater.cc:503] Master 127.13.229.126:36295 was elected leader, sending a full tablet report...
I20260808 06:01:30.972481 26568 ts_tablet_manager.cc:1428] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260808 06:01:30.984066 26409 tablet.cc:2369] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 06:01:31 dist-test-slave-zkpd krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168890, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
W20260808 06:01:31.100539 26547 tablet.cc:2369] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:01:31.151283 26266 tablet.cc:2369] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:01:31.215906 26572 raft_consensus.cc:482] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:01:31.216176 26572 raft_consensus.cc:504] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:31.217695 26572 leader_election.cc:288] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2971c2ede204cee8b7d85e263e1c347 (127.13.229.65:46353), d3c738e04044481499ffd42d5c05e7be (127.13.229.67:38703)
I20260808 06:01:31.224296 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.219059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:44973 (local address 127.13.229.65:46353)
0808 06:01:31.219189 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:31.219194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:31.219564 (+ 370us) server_negotiation.cc:407] Connection header received
0808 06:01:31.219716 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:31.219722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:31.219776 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:31.219864 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:01:31.220302 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.221104 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.222816 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.223416 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.223512 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:31.223563 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:31.224054 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 06:01:31.224133 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:01:31.224491 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.219209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:45009 (local address 127.13.229.67:38703)
0808 06:01:31.219317 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:31.219321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:31.219954 (+ 633us) server_negotiation.cc:407] Connection header received
0808 06:01:31.220140 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:31.220144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:31.220195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:31.220250 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:01:31.220645 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.221339 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.222806 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.223430 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.223512 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:31.223563 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:31.224266 (+ 703us) server_negotiation.cc:299] Negotiation successful
0808 06:01:31.224363 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 06:01:31.224886 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.218597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:38703 (local address 127.13.229.66:45009)
0808 06:01:31.219866 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:31.219878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:31.220019 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:31.220398 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:31.220401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:31.220418 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:01:31.220521 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:31.220526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.221488 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.221491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:31.222662 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:31.222668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.223682 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.223685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:31.224129 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:31.224154 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:01:31.224656 (+ 502us) client_negotiation.cc:240] Negotiation successful
0808 06:01:31.224745 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"mutex_wait_us":91,"thread_start_us":100,"threads_started":1}
I20260808 06:01:31.225102 26577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.218145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46353 (local address 127.13.229.66:44973)
0808 06:01:31.219477 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:31.219492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:31.219616 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:31.220018 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:01:31.220022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:01:31.220036 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:01:31.220170 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:31.220177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.221240 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.221243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:31.222657 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:01:31.222668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.223662 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.223667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:01:31.223879 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:31.223926 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 06:01:31.224774 (+ 848us) client_negotiation.cc:240] Negotiation successful
0808 06:01:31.224844 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":142,"thread_start_us":112,"threads_started":1}
I20260808 06:01:31.225620 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b457587b3efd47f58ce19171dd4a6258" candidate_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c738e04044481499ffd42d5c05e7be" is_pre_election: true
I20260808 06:01:31.225746 26217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b457587b3efd47f58ce19171dd4a6258" candidate_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2971c2ede204cee8b7d85e263e1c347" is_pre_election: true
I20260808 06:01:31.225960 26500 raft_consensus.cc:2454] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6321ef11ad4dc0a9d0c52b9284f42a in term 0.
I20260808 06:01:31.225975 26217 raft_consensus.cc:2454] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6321ef11ad4dc0a9d0c52b9284f42a in term 0.
I20260808 06:01:31.226462 26291 leader_election.cc:302] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6321ef11ad4dc0a9d0c52b9284f42a, a2971c2ede204cee8b7d85e263e1c347; no voters:
I20260808 06:01:31.226893 26572 raft_consensus.cc:2781] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:01:31.226951 26572 raft_consensus.cc:482] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:01:31.226979 26572 raft_consensus.cc:3037] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:01:31.228030 26572 raft_consensus.cc:504] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:31.228354 26572 leader_election.cc:288] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [CANDIDATE]: Term 1 election: Requested vote from peers a2971c2ede204cee8b7d85e263e1c347 (127.13.229.65:46353), d3c738e04044481499ffd42d5c05e7be (127.13.229.67:38703)
I20260808 06:01:31.228917 26217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b457587b3efd47f58ce19171dd4a6258" candidate_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2971c2ede204cee8b7d85e263e1c347"
I20260808 06:01:31.228917 26500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b457587b3efd47f58ce19171dd4a6258" candidate_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3c738e04044481499ffd42d5c05e7be"
I20260808 06:01:31.229032 26217 raft_consensus.cc:3037] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:01:31.229032 26500 raft_consensus.cc:3037] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:01:31.230261 26500 raft_consensus.cc:2454] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6321ef11ad4dc0a9d0c52b9284f42a in term 1.
I20260808 06:01:31.230261 26217 raft_consensus.cc:2454] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6321ef11ad4dc0a9d0c52b9284f42a in term 1.
I20260808 06:01:31.230612 26293 leader_election.cc:302] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6321ef11ad4dc0a9d0c52b9284f42a, d3c738e04044481499ffd42d5c05e7be; no voters:
I20260808 06:01:31.230792 26572 raft_consensus.cc:2781] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:01:31.231062 26572 raft_consensus.cc:686] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 1 LEADER]: Becoming Leader. State: Replica: 4a6321ef11ad4dc0a9d0c52b9284f42a, State: Running, Role: LEADER
I20260808 06:01:31.231385 26572 consensus_queue.cc:237] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } }
I20260808 06:01:31.234400 26037 catalog_manager.cc:5495] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6321ef11ad4dc0a9d0c52b9284f42a (127.13.229.66). New cstate: current_term: 1 leader_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a6321ef11ad4dc0a9d0c52b9284f42a" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 } health_report { overall_health: UNKNOWN } } }
I20260808 06:01:31.273612 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.268655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.13.229.66:33243)
0808 06:01:31.269018 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0808 06:01:31.269023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:01:31.269037 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:01:31.269123 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:01:31.269127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:01:31.269200 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:01:31.269329 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:01:31.269743 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.270538 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.271827 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:01:31.272186 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:01:31.272293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:01:31.272735 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:01:31.273159 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:01:31.273259 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:01:31.273389 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 06:01:31.273445 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260808 06:01:31.279681 26500 raft_consensus.cc:1261] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 1 FOLLOWER]: Refusing update from remote peer 4a6321ef11ad4dc0a9d0c52b9284f42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:01:31.280478 26578 consensus_queue.cc:1035] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3c738e04044481499ffd42d5c05e7be" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:01:31.280691 26217 raft_consensus.cc:1261] T b457587b3efd47f58ce19171dd4a6258 P a2971c2ede204cee8b7d85e263e1c347 [term 1 FOLLOWER]: Refusing update from remote peer 4a6321ef11ad4dc0a9d0c52b9284f42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:01:31.281261 26578 consensus_queue.cc:1035] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2971c2ede204cee8b7d85e263e1c347" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:01:31.290532 26589 mvcc.cc:205] Tried to move back new op lower bound from 7316147778673672192 to 7316147778486521856. Current Snapshot: MvccSnapshot[applied={T|T < 7316147778673672192}]
I20260808 06:01:31.291042 26588 mvcc.cc:205] Tried to move back new op lower bound from 7316147778673672192 to 7316147778486521856. Current Snapshot: MvccSnapshot[applied={T|T < 7316147778673672192}]
I20260808 06:01:31.299013 26036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:01:31.299237 26036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:01:31.301508 26036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f3b6d143f324e699f3c2fa62bdb9c68: Sending DeleteTablet for 3 replicas of tablet b457587b3efd47f58ce19171dd4a6258
I20260808 06:01:31.302193 26189 tablet_service.cc:1514] Processing DeleteTablet for tablet b457587b3efd47f58ce19171dd4a6258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:01:31 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:33196
I20260808 06:01:31.302512 26462 tablet_service.cc:1514] Processing DeleteTablet for tablet b457587b3efd47f58ce19171dd4a6258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:01:31 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:58946
I20260808 06:01:31.302704 26340 tablet_service.cc:1514] Processing DeleteTablet for tablet b457587b3efd47f58ce19171dd4a6258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:01:31 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:41410
I20260808 06:01:31.302953 26600 tablet_replica.cc:330] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: stopping tablet replica
I20260808 06:01:31.303419 26600 raft_consensus.cc:2229] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:01:31.303478 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 26122
I20260808 06:01:31.304288 26600 raft_consensus.cc:2258] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:01:31.305950 26600 ts_tablet_manager.cc:1905] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:01:31.309818 26600 ts_tablet_manager.cc:1918] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:01:31.309999 26600 log.cc:1196] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/b457587b3efd47f58ce19171dd4a6258
I20260808 06:01:31.310429 26600 ts_tablet_manager.cc:1939] T b457587b3efd47f58ce19171dd4a6258 P d3c738e04044481499ffd42d5c05e7be: Deleting consensus metadata
I20260808 06:01:31.311800 26021 catalog_manager.cc:4841] TS d3c738e04044481499ffd42d5c05e7be (127.13.229.67:38703): tablet b457587b3efd47f58ce19171dd4a6258 (table test-table [id=d81c244a5c4b41ff973b19d0b0b508bd]) successfully deleted
W20260808 06:01:31.313684 26019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:46353 (error 108)
I20260808 06:01:31.314662 26601 tablet_replica.cc:330] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a: stopping tablet replica
I20260808 06:01:31.315002 26601 raft_consensus.cc:2229] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:01:31.315373 26601 raft_consensus.cc:2258] T b457587b3efd47f58ce19171dd4a6258 P 4a6321ef11ad4dc0a9d0c52b9284f42a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 06:01:31.316059 26561 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:01:31.314468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:46353 (local address 127.0.0.1:33200)
0808 06:01:31.314590 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:31.314607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:01:31.314699 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:01:31.315930 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:46353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:01:31.316196 26019 catalog_manager.cc:4584] TS a2971c2ede204cee8b7d85e263e1c347 (127.13.229.65:46353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b457587b3efd47f58ce19171dd4a6258: Network error: Client connection negotiation failed: client connection to 127.13.229.65:46353: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:01:31.316579 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 26271
I20260808 06:01:31.323599 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 26413
I20260808 06:01:31.326336 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:01:31.323840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33243 (local address 127.0.0.1:41426)
0808 06:01:31.326163 (+ 2323us) negotiation.cc:107] Waiting for socket to connect
0808 06:01:31.326246 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.66:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2230}
I20260808 06:01:31.331408 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 26000
I20260808 06:01:31.339741 14229 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 06:01:31.684047 14229 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 06:01:31.684216 14229 mini_ranger.cc:66] Stopping Ranger...
I20260808 06:01:32.029237 14229 mini_ranger.cc:68] Stopped Ranger
2026-08-08 06:01:32.029 UTC [25560] LOG: received smart shutdown request
2026-08-08 06:01:32.033 UTC [25560] LOG: background worker "logical replication launcher" (PID 25569) exited with exit code 1
2026-08-08 06:01:32.033 UTC [25564] LOG: shutting down
2026-08-08 06:01:32.180 UTC [25560] LOG: database system is shut down
2026-08-08T06:01:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46090 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:01:32 dist-test-slave-zkpd krb5kdc[26613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:01:32 dist-test-slave-zkpd krb5kdc[26613](info): set up 2 sockets
Aug 08 06:01:32 dist-test-slave-zkpd krb5kdc[26613](info): commencing operation
krb5kdc: starting...
W20260808 06:01:33.396272 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:01:33 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:01:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangeradmin_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangeradmin_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangerlookup_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangerlookup_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
I20260808 06:01:33.564723 14229 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNODW_f/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/postgres -l logfile start
2026-08-08 06:01:35.987 UTC [26645] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 06:01:35.987 UTC [26645] LOG: listening on IPv4 address "127.13.229.84", port 42029
2026-08-08 06:01:35.990 UTC [26645] LOG: listening on Unix socket "/tmp/.s.PGSQL.42029"
2026-08-08 06:01:36.012 UTC [26648] LOG: database system was shut down at 2026-08-08 06:01:34 UTC
2026-08-08 06:01:36.015 UTC [26645] LOG: database system is ready to accept connections
I20260808 06:01:36.993548 14229 mini_postgres.cc:96] Postgres bound to 42029
2026-08-08 06:01:37.001 UTC [26656] FATAL: database "slave" does not exist
127.13.229.84:42029 - accepting connections
I20260808 06:01:37.001597 14229 mini_ranger.cc:161] Starting Ranger...
I20260808 06:01:37.017603 14229 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 06:01:37.511204 14229 mini_ranger.cc:87] Created ranger Postgres database
I20260808 06:01:37.511314 14229 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin
2026-08-08 06:01:37,799 [I] DB FLAVOR :POSTGRES
2026-08-08 06:01:37,799 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:01:37,799 [I] Checking connection..
2026-08-08 06:01:37,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 06:01:38,060 [I] Checking connection passed.
2026-08-08 06:01:38,061 [I] --------- Verifying version history table ---------
2026-08-08 06:01:38,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:01:38,327 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:01:38,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:01:38,594 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 06:01:38,594 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 06:01:38,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 06:01:38.846 UTC [26724] WARNING: there is no transaction in progress
2026-08-08 06:01:38,853 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 06:01:38,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 06:01:39,109 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 06:01:39,109 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:01:39,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 06:01:39,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 06:01:39,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zkpd.c.gcp-upstream.internal','N') ;"
2026-08-08 06:01:39,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:01:40,101 [I] Table x_portal_user does not exist in database ranger
2026-08-08 06:01:40,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:01:40,364 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 06:01:40,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:01:40,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:01:40,856 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 06:01:40,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 06:01:41.806 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.822 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.839 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.854 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.869 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.888 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.952 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.958 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.972 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.982 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:41.992 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42.005 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42.013 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42.019 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42.026 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42.032 UTC [26838] WARNING: there is no transaction in progress
2026-08-08 06:01:42,640 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 06:01:42,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 06:01:42,890 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 06:01:42,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 06:01:43,120 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 06:01:43,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 06:01:43,357 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 06:01:43,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 06:01:43,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 06:01:43,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zkpd.c.gcp-upstream.internal';"
2026-08-08 06:01:44,085 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 06:01:44,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 06:01:44,315 [I] DB_PATCHES have already been applied
I20260808 06:01:44.321825 14229 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNODW_f/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 06:01:44.321895 14229 mini_ranger.cc:193] Using host: 127.13.229.84
I20260808 06:01:44.326320 14229 mini_ranger.cc:227] Ranger admin URL: http://127.13.229.84:51001
Aug 08, 2026 6:01:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:01:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:01:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:01:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:01:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:01:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:01:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 06:01:45.469552 14229 mini_ranger.cc:160] Time spent starting Ranger: real 8.468s user 0.003s sys 0.009s
Aug 08, 2026 6:02:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:02:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:02:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:02:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:02:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 06:02:07.768909 14229 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.229.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.229.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangerkms_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.229.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/rangerkms_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.229.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.229.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.229.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/HTTP_127.13.229.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 06:02:07 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168927, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 06:02:07.863363 14229 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 06:02:07.873633 14229 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 06:02:08.683647 14229 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 06:02:08.683719 14229 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms
I20260808 06:02:08.683733 14229 mini_ranger_kms.cc:227] Using postgres at 127.13.229.84:42029
2026-08-08 06:02:08,862 [I] DB FLAVOR :POSTGRES
2026-08-08 06:02:08,862 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 06:02:08,862 [I] Checking connection
2026-08-08 06:02:08,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 06:02:09,134 [I] connection success
2026-08-08 06:02:09,134 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 06:02:09,134 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 06:02:09,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 06:02:09,394 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 06:02:09,395 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 06:02:09,395 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 06:02:09,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.229.84:42029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 06:02:09,663 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 06:02:10.137200 14229 mini_ranger_kms.cc:310] Created kms service
I20260808 06:02:10.307385 14229 mini_ranger_kms.cc:326] Created kudu user
I20260808 06:02:10.352927 14229 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 06:02:10 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168927, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:10.771973 14229 mini_ranger_kms.cc:384] Added ranger policy
I20260808 06:02:10.772241 14229 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNODW_f/build/debug/bin/postgresql.jar:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNODW_f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 06:02:10.772264 14229 mini_ranger_kms.cc:240] Using host: 127.13.229.84
I20260808 06:02:10.777033 14229 mini_ranger_kms.cc:276] Ranger KMS PID: 27025
I20260808 06:02:10.777128 14229 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.229.84:56043
Aug 08, 2026 6:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:02:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:02:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 06:02:12 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786168932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:13.087035 14229 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.224s user 0.013s sys 0.151s
I20260808 06:02:13.087167 14229 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 06:02:13.087239 14229 mini_ranger_kms.cc:401] 127.13.229.84:56043/kms/v1/keys
Aug 08 06:02:13 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
Aug 08 06:02:13 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168932, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.229.84@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
Aug 08 06:02:16 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:02:16.089076 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41605
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:41605
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:16.226169 27079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:16.226538 27079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:16.226641 27079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:16.230659 27079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:16.230803 27079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:16.230865 27079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:16.230903 27079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:16.230937 27079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:16.236439 27079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.229.84:56043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:41605
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:16.237447 27079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:16.238468 27079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:16.248200 27085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:16.248279 27084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:16.248332 27079 server_base.cc:1034] running on GCE node
W20260808 06:02:16.248495 27087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:16.249065 27079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:16.250154 27079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:16.251349 27079 hybrid_clock.cc:648] HybridClock initialized: now 1786168936251331 us; error 37 us; skew 500 ppm
Aug 08 06:02:16 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:16.254237 27079 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:16.255234 27079 webserver.cc:460] Webserver started at http://127.13.229.126:36505/ using document root <none> and password file <none>
I20260808 06:02:16.255790 27079 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:16.255928 27079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:16.256179 27079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:02:16 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:16.308039 27079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b"
format_stamp: "Formatted at 2026-08-08 06:02:16 on dist-test-slave-zkpd"
server_key: "6e435e35418864e223877c26ed0917ad"
server_key_iv: "a84d822a4c2ac5aa3149e0ae06b6467b"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:16.308718 27079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b"
format_stamp: "Formatted at 2026-08-08 06:02:16 on dist-test-slave-zkpd"
server_key: "6e435e35418864e223877c26ed0917ad"
server_key_iv: "a84d822a4c2ac5aa3149e0ae06b6467b"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:16.312875 27079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 06:02:16.330276 27094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:16.331740 27079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:02:16.331921 27079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b"
format_stamp: "Formatted at 2026-08-08 06:02:16 on dist-test-slave-zkpd"
server_key: "6e435e35418864e223877c26ed0917ad"
server_key_iv: "a84d822a4c2ac5aa3149e0ae06b6467b"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:16.332062 27079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:16.350092 27079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:16.353282 27079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:16.353502 27079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:16.361829 27079 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:41605
I20260808 06:02:16.361840 27148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:41605 every 8 connection(s)
I20260808 06:02:16.362813 27079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:16.365716 27150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:16.367836 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27079
I20260808 06:02:16.368005 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:02:16.368330 14229 external_mini_cluster.cc:1428] Setting key 4469741f6ba24ec809ad560cc7233d87
I20260808 06:02:16.372653 27150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Bootstrap starting.
I20260808 06:02:16.375355 27150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:16.376272 27150 log.cc:824] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:16.378338 27150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: No bootstrap required, opened a new log
Aug 08 06:02:16 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:16.382817 27150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:16.383153 27150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:16.383251 27150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd2f5e3a4a841e59709bc0a9f9c820b, State: Initialized, Role: FOLLOWER
I20260808 06:02:16.383852 27150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:16.384147 27150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:16.384294 27150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:16.384460 27150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:16.385423 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:16.369882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.13.229.126:41605)
0808 06:02:16.370387 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:16.370408 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:16.370455 (+ 47us) server_negotiation.cc:407] Connection header received
0808 06:02:16.371283 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:16.371306 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:16.371764 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:16.372175 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:16.373096 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:16.374592 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:16.375822 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:16.376567 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:16.376754 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:16.379774 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:16.379780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:16.379797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:16.379827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:16.381901 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:16.382407 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:16.382414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:16.382419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:16.382552 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:16.382905 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:16.382908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:16.382910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:16.383434 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:16.383629 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:16.383983 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 06:02:16.384462 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":144,"threads_started":1}
I20260808 06:02:16.385627 27150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:16.386014 27150 leader_election.cc:302] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fd2f5e3a4a841e59709bc0a9f9c820b; no voters:
I20260808 06:02:16.386387 27150 leader_election.cc:288] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:02:16.386489 27155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:16.387764 27155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 LEADER]: Becoming Leader. State: Replica: 1fd2f5e3a4a841e59709bc0a9f9c820b, State: Running, Role: LEADER
I20260808 06:02:16.388180 27155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:16.388675 27150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:16.389415 27155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } } }
I20260808 06:02:16.389609 27155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:16.389868 27155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fd2f5e3a4a841e59709bc0a9f9c820b. Latest consensus state: current_term: 1 leader_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } } }
I20260808 06:02:16.389968 27155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:16.390224 27164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:16.393462 27164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:16.393934 27150 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 06:02:16.405712 27164 catalog_manager.cc:1347] Generated new cluster ID: 6b755b06bc8941f1a835e2de46565003
I20260808 06:02:16.405844 27164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:16.435477 27164 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:02:16.437048 27164 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:16.445289 27164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Generated new TSK 0
I20260808 06:02:16.446237 27164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:02:17.207721 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:17.369431 27195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:17.369755 27195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:17.369872 27195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:17.374660 27195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:17.374816 27195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:17.374958 27195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:17.382867 27195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:17.384595 27195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:17.386060 27195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:17.399477 27204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:17.400060 27206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:17.400372 27203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:17.401599 27195 server_base.cc:1034] running on GCE node
I20260808 06:02:17.402097 27195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:17.402891 27195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:17.404116 27195 hybrid_clock.cc:648] HybridClock initialized: now 1786168937404051 us; error 81 us; skew 500 ppm
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:17.407801 27195 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:17.408893 27195 webserver.cc:460] Webserver started at http://127.13.229.65:39483/ using document root <none> and password file <none>
I20260808 06:02:17.409631 27195 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:17.409722 27195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:17.409971 27195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:17.429903 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "805165669da14ff691eeb86592098c56"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "fe2da7b745993742f50f098d3519f0e0"
server_key_iv: "4178b2c2afece64a7e8ab22f0c706e59"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.430835 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "805165669da14ff691eeb86592098c56"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "fe2da7b745993742f50f098d3519f0e0"
server_key_iv: "4178b2c2afece64a7e8ab22f0c706e59"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.438458 27195 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 06:02:17.472420 27216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.474802 27195 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.000s
I20260808 06:02:17.475014 27195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "805165669da14ff691eeb86592098c56"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "fe2da7b745993742f50f098d3519f0e0"
server_key_iv: "4178b2c2afece64a7e8ab22f0c706e59"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.475283 27195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:17.534861 27195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:17.539033 27195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:17.539309 27195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:17.540071 27195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:17.541246 27195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:17.541355 27195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.541471 27195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:17.541543 27195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.555694 27331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:39469 every 8 connection(s)
I20260808 06:02:17.555768 27195 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:39469
I20260808 06:02:17.557195 27195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:17.566066 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27195
I20260808 06:02:17.566223 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:02:17.566577 14229 external_mini_cluster.cc:1428] Setting key d4078d9d6fb31d68df2523a71f33daca
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:17.575579 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:17.559783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:33891 (local address 127.13.229.126:41605)
0808 06:02:17.560111 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:17.560116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:17.561089 (+ 973us) server_negotiation.cc:407] Connection header received
0808 06:02:17.562425 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:17.562429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:17.562497 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:17.562572 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:17.564383 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.565226 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.567040 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.567423 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.567512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:17.570755 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:17.570762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:17.570767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:17.570802 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:17.572887 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:17.573564 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:17.573568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:17.573570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:17.573669 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:17.574141 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:17.574145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:17.574147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:17.574408 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:17.574503 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:17.575250 (+ 747us) server_negotiation.cc:299] Negotiation successful
0808 06:02:17.575395 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
I20260808 06:02:17.577221 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:17.560308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.65:33891)
0808 06:02:17.560904 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:17.560970 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:17.562138 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:17.563035 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:17.563044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:17.563435 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:17.564182 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:17.564204 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.565412 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.565415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:17.566888 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:17.566901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.567742 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.567746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:17.567992 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:17.568722 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:17.568745 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:17.570550 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:17.573030 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:17.573036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:17.573054 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:17.573430 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:17.573815 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:17.573818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:17.573820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:17.574012 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:17.574579 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:17.574586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:17.574921 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 06:02:17.576060 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0808 06:02:17.576432 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":135,"threads_started":1}
I20260808 06:02:17.578722 27332 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:17.579011 27332 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:02:17.579664 27332 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:17.581601 27112 ts_manager.cc:194] Registered new tserver with Master: 805165669da14ff691eeb86592098c56 (127.13.229.65:39469)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:02:17.583173 27112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:33891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:02:17.602583 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:17.756196 27339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:17.756459 27339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:17.756501 27339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:17.760488 27339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:17.760541 27339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:17.760597 27339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:17.766289 27339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:17.767772 27339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:17.769176 27339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:17.783094 27345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:17.783423 27346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:17.783558 27348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:17.783761 27339 server_base.cc:1034] running on GCE node
I20260808 06:02:17.784267 27339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:17.785070 27339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:17.786258 27339 hybrid_clock.cc:648] HybridClock initialized: now 1786168937786191 us; error 84 us; skew 500 ppm
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:17.795068 27339 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:17.796267 27339 webserver.cc:460] Webserver started at http://127.13.229.66:38315/ using document root <none> and password file <none>
I20260808 06:02:17.797082 27339 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:17.797236 27339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:17.797536 27339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:17.816852 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "540dd6eb467346aa8b2c044cc0260f57"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "606f5b2269fc74d8510a67ebddafe78f"
server_key_iv: "ef75de44176dc0478197b616865f5287"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.817616 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "540dd6eb467346aa8b2c044cc0260f57"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "606f5b2269fc74d8510a67ebddafe78f"
server_key_iv: "ef75de44176dc0478197b616865f5287"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.823962 27339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 06:02:17.852373 27355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.854476 27339 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260808 06:02:17.854724 27339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "540dd6eb467346aa8b2c044cc0260f57"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "606f5b2269fc74d8510a67ebddafe78f"
server_key_iv: "ef75de44176dc0478197b616865f5287"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:17.855005 27339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:17.878582 27339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:17.882730 27339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:17.883020 27339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:17.883956 27339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:17.885179 27339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:17.885288 27339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.885394 27339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:17.885460 27339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:17.897763 27339 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:40579
I20260808 06:02:17.899053 27339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:17.901744 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27339
I20260808 06:02:17.901877 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:02:17.902217 14229 external_mini_cluster.cc:1428] Setting key 4a45710843d65ef27b204dc1f785cda5
I20260808 06:02:17.912819 27478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:40579 every 8 connection(s)
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:02:17.929001 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:17.912081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:36825 (local address 127.13.229.126:41605)
0808 06:02:17.912204 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:17.912209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:17.913879 (+ 1670us) server_negotiation.cc:407] Connection header received
0808 06:02:17.915471 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:17.915478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:17.915548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:17.915613 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:17.917607 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.918512 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.920389 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.920720 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.920801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:17.924484 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:17.924489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:17.924491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:17.924519 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:17.926355 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:17.927052 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:17.927058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:17.927063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:17.927173 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:17.927607 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:17.927610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:17.927612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:17.927833 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:17.927978 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:17.928673 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 06:02:17.928814 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 06:02:17.930934 27483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:17.911394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.66:36825)
0808 06:02:17.913650 (+ 2256us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:17.913717 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:17.915177 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:17.916141 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:17.916152 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:17.916594 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:17.917349 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:17.917374 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.918696 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.918700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:17.920251 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:17.920262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:17.921029 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:17.921033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:17.921235 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:17.921948 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:17.921971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:17.924282 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:17.926492 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:17.926502 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:17.926522 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:17.926913 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:17.927304 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:17.927306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:17.927308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:17.927488 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:17.928048 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:17.928054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:17.928379 (+ 325us) client_negotiation.cc:769] Sending connection context
0808 06:02:17.929468 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0808 06:02:17.929837 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"thread_start_us":157,"threads_started":1}
I20260808 06:02:17.932641 27480 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:17.932907 27480 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:17.933514 27480 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:17.934770 27112 ts_manager.cc:194] Registered new tserver with Master: 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579)
I20260808 06:02:17.935498 27112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:36825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:02:17.960516 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:02:17 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
W20260808 06:02:18.156543 27485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:18.156978 27485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:18.157085 27485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:18.162861 27485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:18.162986 27485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:18.163118 27485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:18.170637 27485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:18.172205 27485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:18.173543 27485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:18.186201 27493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.186203 27491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.186717 27490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:18.187850 27485 server_base.cc:1034] running on GCE node
I20260808 06:02:18.188324 27485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:18.189091 27485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:18.190243 27485 hybrid_clock.cc:648] HybridClock initialized: now 1786168938190175 us; error 83 us; skew 500 ppm
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:18.194024 27485 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:18.195032 27485 webserver.cc:460] Webserver started at http://127.13.229.67:39211/ using document root <none> and password file <none>
I20260808 06:02:18.195705 27485 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:18.195776 27485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:18.196112 27485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:18.213894 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "9f630c7525cd453d89b415567d4a2d87"
format_stamp: "Formatted at 2026-08-08 06:02:18 on dist-test-slave-zkpd"
server_key: "92967d1888155c29d252d9459a8e5cfb"
server_key_iv: "c42d909158f60959a5e09f67af3292ab"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:18.215217 27485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "9f630c7525cd453d89b415567d4a2d87"
format_stamp: "Formatted at 2026-08-08 06:02:18 on dist-test-slave-zkpd"
server_key: "92967d1888155c29d252d9459a8e5cfb"
server_key_iv: "c42d909158f60959a5e09f67af3292ab"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:18.220705 27485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 06:02:18.242483 27502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.244273 27485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 06:02:18.244413 27485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "9f630c7525cd453d89b415567d4a2d87"
format_stamp: "Formatted at 2026-08-08 06:02:18 on dist-test-slave-zkpd"
server_key: "92967d1888155c29d252d9459a8e5cfb"
server_key_iv: "c42d909158f60959a5e09f67af3292ab"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:18.244542 27485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:18.271991 27485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:18.275935 27485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:18.276203 27485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:18.276937 27485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:18.278043 27485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:18.278151 27485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.278249 27485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:18.278311 27485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.292819 27485 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:42141
I20260808 06:02:18.294220 27485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:02:18.300421 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27485
I20260808 06:02:18.300551 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:02:18.300863 14229 external_mini_cluster.cc:1428] Setting key b8bc5732a23f7603f878f36fb0a476d1
I20260808 06:02:18.295895 27619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:42141 every 8 connection(s)
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:18.314611 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:18.299444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:39031 (local address 127.13.229.126:41605)
0808 06:02:18.299591 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:18.299596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:18.300479 (+ 883us) server_negotiation.cc:407] Connection header received
0808 06:02:18.301821 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:18.301825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:18.301881 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:18.301934 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:18.303732 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.304554 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.306232 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.306575 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.306656 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:18.310125 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:18.310129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:18.310132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:18.310160 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:18.312037 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:18.312685 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:18.312689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:18.312691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:18.312780 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:18.313195 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:18.313198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:18.313200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:18.313445 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:18.313530 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:18.314301 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 06:02:18.314435 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:02:18.316485 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:18.298267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.67:39031)
0808 06:02:18.300282 (+ 2015us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:18.300346 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:18.301561 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:18.302375 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:18.302385 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:18.302782 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:18.303483 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:18.303505 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.304726 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.304729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:18.306091 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:18.306102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.306866 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.306870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:18.307071 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:18.307788 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:18.307809 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:18.309873 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:18.312184 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:18.312191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:18.312208 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:18.312559 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:18.312895 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:18.312897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:18.312899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:18.313076 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:18.313618 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:18.313625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:18.314000 (+ 375us) client_negotiation.cc:769] Sending connection context
0808 06:02:18.315041 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0808 06:02:18.315379 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"thread_start_us":179,"threads_started":1}
I20260808 06:02:18.318077 27620 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:18.318336 27620 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:18.318912 27620 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:18.320093 27112 ts_manager.cc:194] Registered new tserver with Master: 9f630c7525cd453d89b415567d4a2d87 (127.13.229.67:42141)
I20260808 06:02:18.320827 27112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:39031
I20260808 06:02:18.326175 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:02:18.327090 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27195
I20260808 06:02:18.349081 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27339
I20260808 06:02:18.359624 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27485
I20260808 06:02:18.368160 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27079
I20260808 06:02:18.376282 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41605
--webserver_interface=127.13.229.126
--webserver_port=36505
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:41605
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:18.487097 27624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:18.487398 27624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:18.487481 27624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:18.491524 27624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:18.491607 27624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:18.491645 27624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:18.491698 27624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:18.491737 27624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:18.497442 27624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:41605
--ranger_config_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=36505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:18.498497 27624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:18.499485 27624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:18.507674 27630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.507733 27629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.507745 27632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:18.507962 27624 server_base.cc:1034] running on GCE node
I20260808 06:02:18.508505 27624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:18.509467 27624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:18.510625 27624 hybrid_clock.cc:648] HybridClock initialized: now 1786168938510601 us; error 37 us; skew 500 ppm
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:18.513360 27624 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:18.514205 27624 webserver.cc:460] Webserver started at http://127.13.229.126:36505/ using document root <none> and password file <none>
I20260808 06:02:18.514714 27624 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:18.514786 27624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:18.563223 27624 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 06:02:18.566030 27641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.567577 27624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:02:18.567709 27624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b"
format_stamp: "Formatted at 2026-08-08 06:02:16 on dist-test-slave-zkpd"
server_key: "6e435e35418864e223877c26ed0917ad"
server_key_iv: "a84d822a4c2ac5aa3149e0ae06b6467b"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:18.568444 27624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:18.586956 27624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:18.589979 27624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:18.590127 27624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:18.597318 27624 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:41605
I20260808 06:02:18.597326 27695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:41605 every 8 connection(s)
I20260808 06:02:18.598353 27624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:18.602492 27696 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:02:18.604060 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27624
I20260808 06:02:18.604197 27696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Bootstrap starting.
I20260808 06:02:18.604545 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:39469
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:41605
--webserver_port=39483
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:18.620846 27696 log.cc:824] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:18.627394 27696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:02:18.627786 27696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Bootstrap complete.
I20260808 06:02:18.632122 27696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:18.632529 27696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd2f5e3a4a841e59709bc0a9f9c820b, State: Initialized, Role: FOLLOWER
I20260808 06:02:18.633059 27696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:18.633203 27696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:18.633291 27696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:18.633410 27696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:02:18.635234 27696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:18.635560 27696 leader_election.cc:302] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fd2f5e3a4a841e59709bc0a9f9c820b; no voters:
I20260808 06:02:18.635878 27696 leader_election.cc:288] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:02:18.636026 27700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:02:18.636273 27700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [term 2 LEADER]: Becoming Leader. State: Replica: 1fd2f5e3a4a841e59709bc0a9f9c820b, State: Running, Role: LEADER
I20260808 06:02:18.636570 27700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } }
I20260808 06:02:18.637125 27696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:18.637501 27700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fd2f5e3a4a841e59709bc0a9f9c820b. Latest consensus state: current_term: 2 leader_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } } }
I20260808 06:02:18.637619 27700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:18.638260 27701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd2f5e3a4a841e59709bc0a9f9c820b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41605 } } }
I20260808 06:02:18.638358 27701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:18.638891 27704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:18.639396 27696 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 06:02:18.645776 27704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:18.646534 27704 catalog_manager.cc:1259] Loaded cluster ID: 6b755b06bc8941f1a835e2de46565003
I20260808 06:02:18.646597 27704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:18.649166 27704 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:18.649855 27704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Loaded TSK: 0
I20260808 06:02:18.651083 27704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:02:18.755391 27698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:18.755729 27698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:18.755800 27698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:18.761925 27698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:18.762003 27698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:18.762091 27698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:18.770056 27698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=39483
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:18.771442 27698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:18.772892 27698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:18.785562 27734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.785578 27735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:18.785562 27737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:18.786126 27698 server_base.cc:1034] running on GCE node
I20260808 06:02:18.786772 27698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:18.787597 27698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:18.790891 27698 hybrid_clock.cc:648] HybridClock initialized: now 1786168938790854 us; error 45 us; skew 500 ppm
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:18.794551 27698 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:18.795779 27698 webserver.cc:460] Webserver started at http://127.13.229.65:39483/ using document root <none> and password file <none>
I20260808 06:02:18.796603 27698 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:18.796764 27698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:18.816287 27698 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 06:02:18.821918 27745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.823740 27698 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260808 06:02:18.823920 27698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "805165669da14ff691eeb86592098c56"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "fe2da7b745993742f50f098d3519f0e0"
server_key_iv: "4178b2c2afece64a7e8ab22f0c706e59"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:18.824713 27698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:18.841424 27698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:18.845126 27698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:18.845333 27698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:18.846071 27698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:18.847157 27698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:18.847227 27698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.847283 27698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:18.847316 27698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:18.859304 27698 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:39469
I20260808 06:02:18.860829 27861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:39469 every 8 connection(s)
I20260808 06:02:18.861774 27698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:18.863334 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27698
I20260808 06:02:18.863862 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:40579
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:41605
--webserver_port=38315
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:02:18 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:18.885768 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:18.864467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.65:52269)
0808 06:02:18.864993 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:18.865048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:18.868893 (+ 3845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:18.870842 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:18.870851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:18.871225 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:18.871969 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:18.871993 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.873950 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.873953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:18.875338 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:18.875350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.876620 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.876623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:18.876808 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:18.877538 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:18.877559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:18.879395 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:18.881977 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:18.882003 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:18.882017 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:18.882359 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:18.882791 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:18.882794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:18.882796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:18.882996 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:18.883804 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:18.883810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:18.884128 (+ 318us) client_negotiation.cc:769] Sending connection context
0808 06:02:18.884342 (+ 214us) client_negotiation.cc:240] Negotiation successful
0808 06:02:18.884682 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":141,"threads_started":1}
I20260808 06:02:18.887652 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:18.865949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:52269 (local address 127.13.229.126:41605)
0808 06:02:18.866448 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:18.866472 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:18.866525 (+ 53us) server_negotiation.cc:407] Connection header received
0808 06:02:18.869530 (+ 3005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:18.869546 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:18.869972 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:18.870329 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:18.872192 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.873785 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.875494 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:18.876235 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:18.876390 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:18.879632 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:18.879639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:18.879656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:18.879689 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:18.881832 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:18.882523 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:18.882529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:18.882535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:18.882676 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:18.883116 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:18.883119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:18.883121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:18.883615 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:18.883746 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:18.886387 (+ 2641us) server_negotiation.cc:299] Negotiation successful
0808 06:02:18.886696 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260808 06:02:18.890863 27863 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:18.891113 27863 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:18.891678 27863 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
W20260808 06:02:18.893973 27863 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:41605 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 06:02:18.894083 27863 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:18.894244 27863 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:18.895027 27863 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:18.895176 27863 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
W20260808 06:02:18.895865 27863 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:02:19.012269 27866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:19.012651 27866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:19.012745 27866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:19.018532 27866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:19.018594 27866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:19.018653 27866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:19.024900 27866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=38315
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:19.026005 27866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:19.027133 27866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:19.039939 27872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:19.039849 27875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:19.040246 27873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:19.040366 27866 server_base.cc:1034] running on GCE node
I20260808 06:02:19.040926 27866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:19.041558 27866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:19.042716 27866 hybrid_clock.cc:648] HybridClock initialized: now 1786168939042681 us; error 47 us; skew 500 ppm
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:19.045572 27866 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:19.046555 27866 webserver.cc:460] Webserver started at http://127.13.229.66:38315/ using document root <none> and password file <none>
I20260808 06:02:19.047196 27866 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:19.047286 27866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:19.070035 27866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 06:02:19.073421 27883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.075244 27866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 06:02:19.075394 27866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "540dd6eb467346aa8b2c044cc0260f57"
format_stamp: "Formatted at 2026-08-08 06:02:17 on dist-test-slave-zkpd"
server_key: "606f5b2269fc74d8510a67ebddafe78f"
server_key_iv: "ef75de44176dc0478197b616865f5287"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:19.076206 27866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:19.107883 27866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:19.111732 27866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:19.111972 27866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:19.112787 27866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:19.113886 27866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:19.113951 27866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.114005 27866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:19.114034 27866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.124964 27866 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:40579
I20260808 06:02:19.126227 27866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:19.132251 28000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:40579 every 8 connection(s)
I20260808 06:02:19.132827 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 27866
I20260808 06:02:19.133353 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.229.84:56043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:42141
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:41605
--webserver_port=39211
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:19.146559 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:19.130168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:49671 (local address 127.13.229.126:41605)
0808 06:02:19.130302 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:19.130306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:19.130323 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:02:19.131287 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:19.131291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:19.131349 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:19.131446 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:19.136091 (+ 4645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.136945 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.138907 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.139289 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.139373 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:19.142292 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:19.142296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:19.142299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:19.142326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:19.144202 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.144798 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.144802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.144804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.144885 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.145277 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.145280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.145282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.145495 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:19.145565 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:19.146248 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 06:02:19.146389 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 06:02:19.148598 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:19.128998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.66:49671)
0808 06:02:19.129677 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:19.129758 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:19.131001 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:19.131670 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:19.131680 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:19.132120 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:19.135840 (+ 3720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:19.135874 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.137141 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.137145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:19.138754 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:19.138767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.139583 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.139587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:19.139789 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:19.140515 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:19.140536 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:19.142083 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:19.144343 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:19.144349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:19.144363 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:19.144679 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:19.144993 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:19.144996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:19.144998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:19.145157 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:19.145648 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:19.145654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:19.145952 (+ 298us) client_negotiation.cc:769] Sending connection context
0808 06:02:19.147148 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0808 06:02:19.147488 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":151,"threads_started":1}
I20260808 06:02:19.150161 28001 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:19.150418 28001 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:19.150971 28001 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
W20260808 06:02:19.152837 28001 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:41605 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 06:02:19.152952 28001 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:19.153102 28001 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:19.153923 28001 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:19.154078 28001 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
W20260808 06:02:19.154800 28001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:02:19.276425 28004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:19.276782 28004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:19.276925 28004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:19.282932 28004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:19.283073 28004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:19.283208 28004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:19.290994 28004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.229.84:56043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:42141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=39211
--tserver_master_addrs=127.13.229.126:41605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:19.292651 28004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:19.294090 28004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:19.307552 28012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:19.307555 28014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:19.308044 28011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:19.308522 28004 server_base.cc:1034] running on GCE node
I20260808 06:02:19.309077 28004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:19.309954 28004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:19.311168 28004 hybrid_clock.cc:648] HybridClock initialized: now 1786168939311134 us; error 48 us; skew 500 ppm
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:19.315557 28004 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:19.316843 28004 webserver.cc:460] Webserver started at http://127.13.229.67:39211/ using document root <none> and password file <none>
I20260808 06:02:19.317641 28004 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:19.317773 28004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:19.341286 28004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 06:02:19.344298 28022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.346103 28004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:02:19.346333 28004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "9f630c7525cd453d89b415567d4a2d87"
format_stamp: "Formatted at 2026-08-08 06:02:18 on dist-test-slave-zkpd"
server_key: "92967d1888155c29d252d9459a8e5cfb"
server_key_iv: "c42d909158f60959a5e09f67af3292ab"
server_key_version: "kuduclusterkey@0"
I20260808 06:02:19.347265 28004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:19.396870 28004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:19.400775 28004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:19.401010 28004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:19.401875 28004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:19.403023 28004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:19.403100 28004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.403167 28004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:19.403210 28004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:19.416086 28004 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:42141
I20260808 06:02:19.417580 28004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:02:19.418573 28140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:42141 every 8 connection(s)
I20260808 06:02:19.423270 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28004
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:19.438210 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:19.423928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.13.229.126:41605)
0808 06:02:19.425962 (+ 2034us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:19.425968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:19.425985 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:02:19.426025 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:19.426027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:19.426081 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:19.426166 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:19.428002 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.429172 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.431994 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.432251 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.432380 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:19.433956 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:19.433960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:19.433963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:19.433998 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:19.435822 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.436720 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.436724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.436726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.436818 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.437174 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.437177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.437179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.437389 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:19.437601 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:19.437833 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 06:02:19.437984 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1531,"thread_start_us":103,"threads_started":1}
I20260808 06:02:19.440579 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:19.419617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:37905 (local address 127.13.229.126:41605)
0808 06:02:19.419750 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:19.419754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:19.420814 (+ 1060us) server_negotiation.cc:407] Connection header received
0808 06:02:19.422164 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:19.422167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:19.422244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:19.422343 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:19.423970 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.424897 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.426827 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.427212 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.427297 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:19.435158 (+ 7861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:19.435161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:19.435163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:19.435205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:19.437121 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.438570 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.438574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.438576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.438671 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:19.439330 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:19.439333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:19.439335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:19.439562 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:19.439671 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:19.440366 (+ 695us) server_negotiation.cc:299] Negotiation successful
0808 06:02:19.440467 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:19.441710 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:19.420080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:41605 (local address 127.13.229.67:37905)
0808 06:02:19.420608 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:19.420673 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:19.421901 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:19.422572 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:19.422582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:19.422991 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:19.423707 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:19.423731 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.425088 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.425091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:19.426632 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:19.426644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:19.427517 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:19.427523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:19.427758 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:19.428455 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:19.428473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:19.434753 (+ 6280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:19.437230 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:19.437237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:19.437255 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:19.438440 (+ 1185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:19.438813 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:19.438816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:19.438818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:19.439020 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:19.439690 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:19.439697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:19.440076 (+ 379us) client_negotiation.cc:769] Sending connection context
0808 06:02:19.440291 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 06:02:19.440640 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":106,"threads_started":1}
I20260808 06:02:19.442893 28141 heartbeater.cc:347] Connected to a master server at 127.13.229.126:41605
I20260808 06:02:19.443166 28141 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:19.443854 28141 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:19.445358 27659 ts_manager.cc:194] Registered new tserver with Master: 9f630c7525cd453d89b415567d4a2d87 (127.13.229.67:42141)
I20260808 06:02:19.446799 27659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:37905
Aug 08 06:02:19 dist-test-slave-zkpd krb5kdc[26613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:19.896116 27863 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:19.896452 27863 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:19.897377 27659 ts_manager.cc:194] Registered new tserver with Master: 805165669da14ff691eeb86592098c56 (127.13.229.65:39469)
I20260808 06:02:19.898226 27659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:52269
I20260808 06:02:20.155040 28001 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:20.155362 28001 heartbeater.cc:511] Master 127.13.229.126:41605 requested a full tablet report, sending...
I20260808 06:02:20.156183 27658 ts_manager.cc:194] Registered new tserver with Master: 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579)
I20260808 06:02:20.156826 27658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:49671
Aug 08 06:02:20 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:20.160331 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:02:20.171931 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.161849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.13.229.126:41605)
0808 06:02:20.162092 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.162097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.162249 (+ 152us) server_negotiation.cc:407] Connection header received
0808 06:02:20.162399 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.162404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.162458 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.162560 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:20.163362 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.164211 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.165447 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.165801 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.165906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.167814 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:20.167817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:20.167820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:20.167845 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:20.169742 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:20.170230 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:20.170234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:20.170235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:20.170321 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:20.170606 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:20.170609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:20.170611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:20.170809 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:20.170924 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.171568 (+ 644us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.171695 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260808 06:02:20.182361 27658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:02:20.186847 27658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:02:20.209429 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.200290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:42141 (local address 127.0.0.1:54616)
0808 06:02:20.201991 (+ 1701us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.202045 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.202251 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.203474 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.203478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.203499 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.203610 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.203615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.205072 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.205076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.207358 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.207372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.208846 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.208849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.208980 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.209023 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.209137 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.209267 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"thread_start_us":118,"threads_started":1}
I20260808 06:02:20.210882 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.200437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.13.229.67:42141)
0808 06:02:20.200758 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.200766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.202137 (+ 1371us) server_negotiation.cc:407] Connection header received
0808 06:02:20.202833 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.202843 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.203078 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.203226 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.203733 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.204920 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.207522 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.208478 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.208568 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.208724 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.210601 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.210720 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260808 06:02:20.214493 28171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.201396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39469 (local address 127.0.0.1:35362)
0808 06:02:20.208067 (+ 6671us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.208083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.208202 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.209032 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.209037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.209057 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.209151 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.209157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.210612 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.210616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.212703 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.212716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.214058 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.214062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.214154 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.214183 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.214262 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.214332 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6591,"thread_start_us":105,"threads_started":1}
I20260808 06:02:20.215055 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.201956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.13.229.65:39469)
0808 06:02:20.202616 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.202623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.208322 (+ 5699us) server_negotiation.cc:407] Connection header received
0808 06:02:20.208421 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.208428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.208681 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.208841 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.209290 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.210464 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.212857 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.213681 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.213821 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.213987 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.214787 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.214910 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":71,"threads_started":1}
I20260808 06:02:20.214829 28072 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 (DEFAULT_TABLE table=test-table [id=d13296eb29ec4764831756af4b38fdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:20.216187 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6670f923d94c8c850eb2a8302a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:20.218531 27778 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 (DEFAULT_TABLE table=test-table [id=d13296eb29ec4764831756af4b38fdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:20.219626 27778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6670f923d94c8c850eb2a8302a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:20.228037 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.201661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:40579 (local address 127.0.0.1:47930)
0808 06:02:20.211537 (+ 9876us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.211550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.211615 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.223330 (+ 11715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.223333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.223354 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.223454 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.223460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.225018 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.225022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.226253 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.226264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.227600 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.227604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.227698 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.227725 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.227798 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.227871 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9819,"thread_start_us":91,"threads_started":1}
I20260808 06:02:20.228667 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.204204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.13.229.66:40579)
0808 06:02:20.222661 (+ 18457us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.222669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.222697 (+ 28us) server_negotiation.cc:407] Connection header received
0808 06:02:20.222776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.222781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.223013 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.223172 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.223587 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.224878 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.226453 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.227271 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.227381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.227531 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.228426 (+ 895us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.228528 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18327,"thread_start_us":119,"threads_started":1}
I20260808 06:02:20.230782 28180 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Bootstrap starting.
I20260808 06:02:20.231666 27931 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 (DEFAULT_TABLE table=test-table [id=d13296eb29ec4764831756af4b38fdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:20.232821 27931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6670f923d94c8c850eb2a8302a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:20.234688 28180 tablet_bootstrap.cc:654] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:20.235563 28180 log.cc:824] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:20.239833 28179 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Bootstrap starting.
I20260808 06:02:20.244603 28180 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: No bootstrap required, opened a new log
I20260808 06:02:20.244916 28180 ts_tablet_manager.cc:1397] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260808 06:02:20.249010 28179 tablet_bootstrap.cc:654] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:20.249610 28182 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Bootstrap starting.
I20260808 06:02:20.249964 28179 log.cc:824] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:20.253041 28182 tablet_bootstrap.cc:654] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:20.253880 28182 log.cc:824] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:20.248968 28180 raft_consensus.cc:348] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.254565 28180 raft_consensus.cc:374] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:20.254626 28180 raft_consensus.cc:729] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805165669da14ff691eeb86592098c56, State: Initialized, Role: FOLLOWER
I20260808 06:02:20.255137 28180 consensus_queue.cc:260] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.255860 27863 heartbeater.cc:503] Master 127.13.229.126:41605 was elected leader, sending a full tablet report...
I20260808 06:02:20.256309 28180 ts_tablet_manager.cc:1428] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260808 06:02:20.258340 28182 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: No bootstrap required, opened a new log
I20260808 06:02:20.258608 28182 ts_tablet_manager.cc:1397] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 06:02:20.262113 28179 tablet_bootstrap.cc:492] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: No bootstrap required, opened a new log
I20260808 06:02:20.262342 28179 ts_tablet_manager.cc:1397] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20260808 06:02:20.266311 28179 raft_consensus.cc:348] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.266562 28179 raft_consensus.cc:374] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:20.266629 28179 raft_consensus.cc:729] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f630c7525cd453d89b415567d4a2d87, State: Initialized, Role: FOLLOWER
I20260808 06:02:20.267344 28179 consensus_queue.cc:260] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.268146 28141 heartbeater.cc:503] Master 127.13.229.126:41605 was elected leader, sending a full tablet report...
I20260808 06:02:20.268527 28179 ts_tablet_manager.cc:1428] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:02:20.271517 28182 raft_consensus.cc:348] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.271968 28182 raft_consensus.cc:374] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:20.272106 28182 raft_consensus.cc:729] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540dd6eb467346aa8b2c044cc0260f57, State: Initialized, Role: FOLLOWER
I20260808 06:02:20.272668 28182 consensus_queue.cc:260] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.273424 28001 heartbeater.cc:503] Master 127.13.229.126:41605 was elected leader, sending a full tablet report...
I20260808 06:02:20.275084 28182 ts_tablet_manager.cc:1428] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260808 06:02:20.359922 28186 raft_consensus.cc:482] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:20.360189 28186 raft_consensus.cc:504] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.361660 28186 leader_election.cc:288] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 805165669da14ff691eeb86592098c56 (127.13.229.65:39469), 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579)
W20260808 06:02:20.364094 27864 tablet.cc:2369] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:02:20.367102 28185 raft_consensus.cc:482] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:20.367529 28185 raft_consensus.cc:504] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.368351 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.362369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:35949 (local address 127.13.229.65:39469)
0808 06:02:20.362529 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.362535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.362768 (+ 233us) server_negotiation.cc:407] Connection header received
0808 06:02:20.363027 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.363030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.363083 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.363134 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.363606 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.364340 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.365964 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.366549 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.366658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.366713 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.368133 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.368219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:02:20.372051 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.361871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39469 (local address 127.13.229.67:35949)
0808 06:02:20.362674 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.362690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.362880 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.363295 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.363298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.363317 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.363469 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.363476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.364496 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.364498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.365819 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.365830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.366932 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.367842 (+ 910us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.367961 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.367989 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.371756 (+ 3767us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.371858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":128,"threads_started":1}
I20260808 06:02:20.372790 27802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "9f630c7525cd453d89b415567d4a2d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805165669da14ff691eeb86592098c56" is_pre_election: true
I20260808 06:02:20.373027 27802 raft_consensus.cc:2454] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f630c7525cd453d89b415567d4a2d87 in term 0.
I20260808 06:02:20.373680 28026 leader_election.cc:302] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805165669da14ff691eeb86592098c56, 9f630c7525cd453d89b415567d4a2d87; no voters:
I20260808 06:02:20.374032 28186 raft_consensus.cc:2781] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:02:20.374107 28186 raft_consensus.cc:482] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:02:20.374147 28186 raft_consensus.cc:3037] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:20.375653 28186 raft_consensus.cc:504] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.376075 28186 leader_election.cc:288] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [CANDIDATE]: Term 1 election: Requested vote from peers 805165669da14ff691eeb86592098c56 (127.13.229.65:39469), 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579)
I20260808 06:02:20.376741 27802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "9f630c7525cd453d89b415567d4a2d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "805165669da14ff691eeb86592098c56"
I20260808 06:02:20.376904 27802 raft_consensus.cc:3037] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:20.378216 27802 raft_consensus.cc:2454] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f630c7525cd453d89b415567d4a2d87 in term 1.
I20260808 06:02:20.378602 28026 leader_election.cc:302] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805165669da14ff691eeb86592098c56, 9f630c7525cd453d89b415567d4a2d87; no voters:
I20260808 06:02:20.378832 28186 raft_consensus.cc:2781] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:20.379032 28186 raft_consensus.cc:686] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 1 LEADER]: Becoming Leader. State: Replica: 9f630c7525cd453d89b415567d4a2d87, State: Running, Role: LEADER
I20260808 06:02:20.379392 28186 consensus_queue.cc:237] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } }
I20260808 06:02:20.380055 28185 leader_election.cc:288] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f630c7525cd453d89b415567d4a2d87 (127.13.229.67:42141), 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579)
I20260808 06:02:20.382622 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.362393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54723 (local address 127.13.229.66:40579)
0808 06:02:20.362519 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.362524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.362931 (+ 407us) server_negotiation.cc:407] Connection header received
0808 06:02:20.363142 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.363145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.363198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.363251 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.363749 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.364538 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.366161 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.369009 (+ 2848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.369227 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.369385 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.382337 (+ 12952us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.382442 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:02:20.383155 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.361982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:40579 (local address 127.13.229.67:54723)
0808 06:02:20.362837 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.362852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.363011 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.363407 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.363410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.363431 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.363589 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.363596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.364851 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.364854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.366023 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.366035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.382037 (+ 16002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.382041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.382165 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.382192 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.382885 (+ 693us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.382985 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":117,"threads_started":1}
I20260808 06:02:20.383239 27658 catalog_manager.cc:5495] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f630c7525cd453d89b415567d4a2d87 (127.13.229.67). New cstate: current_term: 1 leader_uuid: "9f630c7525cd453d89b415567d4a2d87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f630c7525cd453d89b415567d4a2d87" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 } health_report { overall_health: UNKNOWN } } }
W20260808 06:02:20.383641 28002 tablet.cc:2369] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:02:20.384563 27934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "9f630c7525cd453d89b415567d4a2d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540dd6eb467346aa8b2c044cc0260f57" is_pre_election: true
I20260808 06:02:20.384884 27934 raft_consensus.cc:2454] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f630c7525cd453d89b415567d4a2d87 in term 0.
I20260808 06:02:20.385236 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "9f630c7525cd453d89b415567d4a2d87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540dd6eb467346aa8b2c044cc0260f57"
I20260808 06:02:20.385346 27951 raft_consensus.cc:3037] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:20.388545 27951 raft_consensus.cc:2454] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f630c7525cd453d89b415567d4a2d87 in term 1.
I20260808 06:02:20.388746 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.371467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:43435 (local address 127.13.229.67:42141)
0808 06:02:20.371581 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.371585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.383715 (+ 12130us) server_negotiation.cc:407] Connection header received
0808 06:02:20.383972 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.383978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.384033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.384084 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.384501 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.385239 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.387211 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.387771 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.387856 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.387935 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.388509 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.388632 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:02:20.389076 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.371079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:42141 (local address 127.13.229.65:43435)
0808 06:02:20.383629 (+ 12550us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.383645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.383810 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.384231 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.384234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.384253 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.384381 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.384386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.385385 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.385389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.387074 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.387085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.388026 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.388029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.388128 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.388150 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.388867 (+ 717us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.388945 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12479,"thread_start_us":126,"threads_started":1}
I20260808 06:02:20.389819 28074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "805165669da14ff691eeb86592098c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f630c7525cd453d89b415567d4a2d87" is_pre_election: true
I20260808 06:02:20.393349 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.380829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48493 (local address 127.13.229.66:40579)
0808 06:02:20.382844 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.382850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.388441 (+ 5591us) server_negotiation.cc:407] Connection header received
0808 06:02:20.389368 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.389371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.389415 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.389456 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:20.389848 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.390537 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.392146 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.392709 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.392806 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.392848 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.393167 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.393234 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":109,"threads_started":1}
I20260808 06:02:20.393704 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.380407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:40579 (local address 127.13.229.65:48493)
0808 06:02:20.388367 (+ 7960us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.388381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:20.389210 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:20.389580 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:20.389583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:20.389597 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:20.389700 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.389705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.390675 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.390678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.391911 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:20.391924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.392939 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.392942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:20.393043 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.393063 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 06:02:20.393515 (+ 452us) client_negotiation.cc:240] Negotiation successful
0808 06:02:20.393584 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7897,"thread_start_us":103,"threads_started":1}
I20260808 06:02:20.394178 27951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6670f923d94c8c850eb2a8302a7034" candidate_uuid: "805165669da14ff691eeb86592098c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540dd6eb467346aa8b2c044cc0260f57" is_pre_election: true
I20260808 06:02:20.394357 27951 raft_consensus.cc:2379] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 805165669da14ff691eeb86592098c56 in current term 1: Already voted for candidate 9f630c7525cd453d89b415567d4a2d87 in this term.
I20260808 06:02:20.394824 27747 leader_election.cc:302] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 805165669da14ff691eeb86592098c56; no voters: 540dd6eb467346aa8b2c044cc0260f57, 9f630c7525cd453d89b415567d4a2d87
I20260808 06:02:20.395105 28185 raft_consensus.cc:2726] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 06:02:20.420047 28142 tablet.cc:2369] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 06:02:20 dist-test-slave-zkpd krb5kdc[26613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168939, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for HTTP/127.13.229.84@KRBTEST.COM
I20260808 06:02:20.449591 28172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.444240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.13.229.67:42141)
0808 06:02:20.444703 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:20.444707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:20.444722 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:02:20.444818 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:20.444822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:20.444890 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:20.445003 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:02:20.445334 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.446095 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.447351 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:20.447670 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:20.447772 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:20.448216 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:02:20.448846 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:02:20.448996 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:20.449387 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 06:02:20.449449 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260808 06:02:20.465482 27951 raft_consensus.cc:1261] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 1 FOLLOWER]: Refusing update from remote peer 9f630c7525cd453d89b415567d4a2d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:20.466262 28186 consensus_queue.cc:1035] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540dd6eb467346aa8b2c044cc0260f57" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:20.471840 27802 raft_consensus.cc:1261] T 8a6670f923d94c8c850eb2a8302a7034 P 805165669da14ff691eeb86592098c56 [term 1 FOLLOWER]: Refusing update from remote peer 9f630c7525cd453d89b415567d4a2d87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:20.472512 28186 consensus_queue.cc:1035] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "805165669da14ff691eeb86592098c56" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:20.490469 28199 mvcc.cc:205] Tried to move back new op lower bound from 7316147980138139648 to 7316147979798843392. Current Snapshot: MvccSnapshot[applied={T|T < 7316147980138139648}]
I20260808 06:02:20.491091 28198 mvcc.cc:205] Tried to move back new op lower bound from 7316147980138139648 to 7316147979798843392. Current Snapshot: MvccSnapshot[applied={T|T < 7316147980138139648}]
I20260808 06:02:20.500022 28201 mvcc.cc:205] Tried to move back new op lower bound from 7316147980138139648 to 7316147979798843392. Current Snapshot: MvccSnapshot[applied={T|T < 7316147980138139648}]
I20260808 06:02:20.509653 27658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:20.509852 27658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:20.512666 27658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fd2f5e3a4a841e59709bc0a9f9c820b: Sending DeleteTablet for 3 replicas of tablet 8a6670f923d94c8c850eb2a8302a7034
I20260808 06:02:20.513960 28072 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:20 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:54616
I20260808 06:02:20.514757 27778 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:20 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:35362
I20260808 06:02:20.515633 27931 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6670f923d94c8c850eb2a8302a7034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:20 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:47930
I20260808 06:02:20.516892 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27698
I20260808 06:02:20.521456 28210 tablet_replica.cc:330] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: stopping tablet replica
I20260808 06:02:20.521842 28210 raft_consensus.cc:2229] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:20.522229 28210 raft_consensus.cc:2258] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:20.522660 28208 tablet_replica.cc:330] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: stopping tablet replica
I20260808 06:02:20.522948 28208 raft_consensus.cc:2229] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:02:20.523381 28208 raft_consensus.cc:2258] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:20.525430 28210 ts_tablet_manager.cc:1905] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:20.528368 28210 ts_tablet_manager.cc:1918] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:20.528477 28210 log.cc:1196] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/8a6670f923d94c8c850eb2a8302a7034
I20260808 06:02:20.528824 28210 ts_tablet_manager.cc:1939] T 8a6670f923d94c8c850eb2a8302a7034 P 540dd6eb467346aa8b2c044cc0260f57: Deleting consensus metadata
I20260808 06:02:20.530108 27643 catalog_manager.cc:4841] TS 540dd6eb467346aa8b2c044cc0260f57 (127.13.229.66:40579): tablet 8a6670f923d94c8c850eb2a8302a7034 (table test-table [id=d13296eb29ec4764831756af4b38fdf4]) successfully deleted
I20260808 06:02:20.536170 28208 ts_tablet_manager.cc:1905] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:20.539023 28208 ts_tablet_manager.cc:1918] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:20.539137 28208 log.cc:1196] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/8a6670f923d94c8c850eb2a8302a7034
I20260808 06:02:20.539529 28208 ts_tablet_manager.cc:1939] T 8a6670f923d94c8c850eb2a8302a7034 P 9f630c7525cd453d89b415567d4a2d87: Deleting consensus metadata
I20260808 06:02:20.543197 27642 catalog_manager.cc:4841] TS 9f630c7525cd453d89b415567d4a2d87 (127.13.229.67:42141): tablet 8a6670f923d94c8c850eb2a8302a7034 (table test-table [id=d13296eb29ec4764831756af4b38fdf4]) successfully deleted
W20260808 06:02:20.544791 27645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:39469 (error 108)
I20260808 06:02:20.545888 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27866
I20260808 06:02:20.549873 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:20.549566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39469 (local address 127.0.0.1:59322)
0808 06:02:20.549697 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:20.549785 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:39469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 06:02:20.550166 27645 catalog_manager.cc:4584] TS 805165669da14ff691eeb86592098c56 (127.13.229.65:39469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a6670f923d94c8c850eb2a8302a7034: Network error: Client connection negotiation failed: client connection to 127.13.229.65:39469: connect: Connection refused (error 111)
I20260808 06:02:20.563078 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28004
I20260808 06:02:20.575002 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 27624
I20260808 06:02:20.584403 14229 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 06:02:20.928977 14229 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 06:02:20.929095 14229 mini_ranger.cc:66] Stopping Ranger...
I20260808 06:02:21.274022 14229 mini_ranger.cc:68] Stopped Ranger
2026-08-08 06:02:21.274 UTC [26645] LOG: received smart shutdown request
2026-08-08 06:02:21.278 UTC [26645] LOG: background worker "logical replication launcher" (PID 26654) exited with exit code 1
2026-08-08 06:02:21.278 UTC [26649] LOG: shutting down
2026-08-08 06:02:21.437 UTC [26645] LOG: database system is shut down
2026-08-08T06:02:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49250 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:02:21 dist-test-slave-zkpd krb5kdc[28221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:02:21 dist-test-slave-zkpd krb5kdc[28221](info): set up 2 sockets
Aug 08 06:02:21 dist-test-slave-zkpd krb5kdc[28221](info): commencing operation
krb5kdc: starting...
W20260808 06:02:22.633323 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:02:22 dist-test-slave-zkpd krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:02:22.756114 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45823
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:22.875789 28234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:22.876199 28234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:22.876327 28234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:22.882197 28234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:22.882332 28234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:22.882400 28234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:22.882450 28234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:22.882499 28234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:22.890187 28234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:22.891784 28234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:22.893179 28234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:22.902662 28242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:22.902676 28239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:22.902663 28240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:22.902865 28234 server_base.cc:1034] running on GCE node
I20260808 06:02:22.903785 28234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:22.905409 28234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:22.906616 28234 hybrid_clock.cc:648] HybridClock initialized: now 1786168942906583 us; error 49 us; skew 500 ppm
Aug 08 06:02:22 dist-test-slave-zkpd krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:22.910517 28234 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:22.911578 28234 webserver.cc:460] Webserver started at http://127.13.229.126:44473/ using document root <none> and password file <none>
I20260808 06:02:22.912317 28234 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:22.912401 28234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:22.912636 28234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:22.915254 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "fcb30655570b4fb587fd597389609d0c"
format_stamp: "Formatted at 2026-08-08 06:02:22 on dist-test-slave-zkpd"
server_key: "a82a296f8a5c6ed7e5ae4cbf2a0a8626"
server_key_iv: "bbbe9b0df468c61053f0c545ce9e7e63"
server_key_version: "encryptionkey@0"
I20260808 06:02:22.915938 28234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "fcb30655570b4fb587fd597389609d0c"
format_stamp: "Formatted at 2026-08-08 06:02:22 on dist-test-slave-zkpd"
server_key: "a82a296f8a5c6ed7e5ae4cbf2a0a8626"
server_key_iv: "bbbe9b0df468c61053f0c545ce9e7e63"
server_key_version: "encryptionkey@0"
I20260808 06:02:22.920455 28234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260808 06:02:22.923077 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:22.924350 28234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:22.924479 28234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "fcb30655570b4fb587fd597389609d0c"
format_stamp: "Formatted at 2026-08-08 06:02:22 on dist-test-slave-zkpd"
server_key: "a82a296f8a5c6ed7e5ae4cbf2a0a8626"
server_key_iv: "bbbe9b0df468c61053f0c545ce9e7e63"
server_key_version: "encryptionkey@0"
I20260808 06:02:22.924580 28234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:22.941495 28234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:22.944861 28234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:22.945082 28234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:22.952462 28234 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:45823
I20260808 06:02:22.952759 28304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:45823 every 8 connection(s)
I20260808 06:02:22.953534 28234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:22.956379 28305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:22.960265 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:22.962713 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28234
I20260808 06:02:22.962899 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:02:22.963232 14229 external_mini_cluster.cc:1428] Setting key 82000345a07644fdcf8466950020ac0c
Aug 08 06:02:22 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.125@KRBTEST.COM" created.
I20260808 06:02:22.974992 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:22.962489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45823 (local address 127.0.0.1:60640)
0808 06:02:22.962955 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:22.963007 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:22.963835 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:22.964745 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:22.964749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:22.965066 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:22.965399 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:22.965412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:22.966565 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:22.966570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:22.967268 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:22.967276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:22.967919 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:22.967924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:22.968049 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:22.968686 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:22.968703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:22.970087 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:22.972041 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:22.972050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:22.972059 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:22.972339 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:22.972678 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:22.972681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:22.972683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:22.972794 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:22.973369 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:22.973375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:22.973516 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 06:02:22.973680 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 06:02:22.974051 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":161,"thread_start_us":132,"threads_started":1}
I20260808 06:02:22.974992 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:22.962490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.13.229.126:45823)
0808 06:02:22.963169 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:22.963196 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:22.963247 (+ 51us) server_negotiation.cc:407] Connection header received
0808 06:02:22.964267 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:22.964276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:22.964524 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:22.964735 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:22.965556 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:22.966441 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:22.967385 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:22.967750 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:22.967943 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:22.970234 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:22.970241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:22.970250 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:22.970269 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:22.971851 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:22.972449 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:22.972455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:22.972457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:22.972577 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:22.972889 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:22.972892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:22.972894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:22.973250 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:22.973387 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:22.973758 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 06:02:22.974000 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":558,"spinlock_wait_cycles":896,"thread_start_us":159,"threads_started":1}
I20260808 06:02:22.977483 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:22.978257 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:22.978000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37398)
0808 06:02:22.978116 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:22.978182 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 06:02:22.978538 28254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
I20260808 06:02:22.979331 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:22.979133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37408)
0808 06:02:22.979229 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:22.979269 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 06:02:22.979507 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:40437: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
I20260808 06:02:22.990947 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:40437
--webserver_interface=127.13.229.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:23.028751 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } attempt: 1
I20260808 06:02:23.029719 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.029392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37420)
0808 06:02:23.029541 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.029641 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 06:02:23.030457 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.030240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37432)
0808 06:02:23.030347 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.030400 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 06:02:23.030586 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:40437: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
I20260808 06:02:23.098724 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } attempt: 2
I20260808 06:02:23.099625 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.099328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37446)
0808 06:02:23.099483 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.099547 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 06:02:23.100448 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.100213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37458)
0808 06:02:23.100324 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.100380 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:02:23.100584 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:40437: Network error: Client connection negotiation failed: client connection to 127.13.229.125:40437: connect: Connection refused (error 111)
W20260808 06:02:23.105859 28313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:23.106164 28313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:23.106276 28313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:23.110803 28313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:23.110885 28313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:23.110948 28313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:23.111006 28313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:23.111045 28313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:23.116858 28313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.229.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:23.117929 28313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:23.118956 28313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:02:23.127787 28313 server_base.cc:1034] running on GCE node
W20260808 06:02:23.127753 28318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:23.127753 28321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:23.127754 28319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:23.128645 28313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:23.129683 28313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:23.130841 28313 hybrid_clock.cc:648] HybridClock initialized: now 1786168943130830 us; error 28 us; skew 500 ppm
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:23.133610 28313 init.cc:377] Logged in from keytab as kudu/127.13.229.125@KRBTEST.COM (short username kudu)
I20260808 06:02:23.134469 28313 webserver.cc:460] Webserver started at http://127.13.229.125:40173/ using document root <none> and password file <none>
I20260808 06:02:23.135022 28313 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:23.135113 28313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:23.135346 28313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:23.137410 28313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data/instance:
uuid: "311234c079c4445a8df9232581baaf38"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "ca4547ca0960e9a31ae7a799dac7c889"
server_key_iv: "f267d6fcd7a17d1db4460b55c9bba768"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.137960 28313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal/instance:
uuid: "311234c079c4445a8df9232581baaf38"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "ca4547ca0960e9a31ae7a799dac7c889"
server_key_iv: "f267d6fcd7a17d1db4460b55c9bba768"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.141543 28313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 06:02:23.143879 28329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:23.145066 28313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:23.145207 28313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "311234c079c4445a8df9232581baaf38"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "ca4547ca0960e9a31ae7a799dac7c889"
server_key_iv: "f267d6fcd7a17d1db4460b55c9bba768"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.145315 28313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:23.157822 28313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:23.161101 28313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:23.161304 28313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:23.168358 28313 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.125:40437
I20260808 06:02:23.168635 28383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.125:40437 every 8 connection(s)
I20260808 06:02:23.169370 28313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
I20260808 06:02:23.172128 28384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:23.175781 28384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:23.177378 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28313
I20260808 06:02:23.177527 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-1/wal/instance
I20260808 06:02:23.177829 14229 external_mini_cluster.cc:1428] Setting key e06f6de0234ac38930cd8db3f0ede2a3
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.124@KRBTEST.COM" created.
I20260808 06:02:23.191342 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.177874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.13.229.126:45823)
0808 06:02:23.178042 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.178047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.178957 (+ 910us) server_negotiation.cc:407] Connection header received
0808 06:02:23.180300 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.180306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.180369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.180466 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.181767 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.182290 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.184196 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.184427 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.184515 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.187618 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.187621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.187624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.187649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.189219 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.189746 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.189750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.189751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.189838 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.190136 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.190139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.190140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.190337 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.190434 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.191041 (+ 607us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.191167 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 06:02:23.191730 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } attempt: 3
I20260808 06:02:23.192854 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.178250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45823 (local address 127.0.0.1:60646)
0808 06:02:23.178756 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.178817 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.180085 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.180623 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.180629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.180973 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.181598 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.181619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.182422 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.182428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.184069 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.184085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.184596 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.184603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.184868 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.185486 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.185508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.187455 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.189324 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.189329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.189340 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.189592 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.189920 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.189922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.189923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.190054 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.190468 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.190476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.190775 (+ 299us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.190977 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.191286 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260808 06:02:23.194433 28384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:02:23.202396 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.192154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37468)
0808 06:02:23.192252 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.192269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.192351 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.192927 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.192930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.193103 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.193687 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.193696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.195373 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.195378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.196172 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.196181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.196866 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.196870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.196980 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.197456 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.197476 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.198725 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.200779 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.200783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.200785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.200986 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.201283 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.201287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.201289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.201386 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.201849 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.201852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.201957 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.202075 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.202211 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 06:02:23.202853 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.192277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.13.229.125:40437)
0808 06:02:23.192588 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.192598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.192618 (+ 20us) server_negotiation.cc:407] Connection header received
0808 06:02:23.192650 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.192654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.192796 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.192952 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.194300 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.195255 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.196283 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.196750 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.196867 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.198860 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.198869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.198883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.198912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.200646 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.201098 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.201104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.201106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.201187 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.201498 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.201503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.201505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.201741 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.201882 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.202166 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.202363 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":209,"thread_start_us":132,"threads_started":1}
I20260808 06:02:23.204810 28305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:23.205555 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.205295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33696)
0808 06:02:23.205434 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.205488 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 06:02:23.206313 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.206038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33706)
0808 06:02:23.206208 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.206255 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260808 06:02:23.206331 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.194948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37480)
0808 06:02:23.195052 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.195070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.195151 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.195632 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.195635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.195800 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.195952 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.195959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.197230 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.197234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.198096 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.198111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.198798 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.198801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.198917 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.199330 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.199349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.202645 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.204729 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.204732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.204734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.205013 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.205264 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.205267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.205269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.205354 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.205820 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.205824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.205928 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.206037 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.206172 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 06:02:23.206449 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.195071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.13.229.125:40437)
0808 06:02:23.195396 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.195407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.195422 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:02:23.195457 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.195460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.195505 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.195596 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.196078 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.197097 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.198264 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.198672 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.199134 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.202813 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.202818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.202821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.202852 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.204619 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.205106 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.205108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.205109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.205168 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.205461 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.205464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.205487 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.205728 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.205881 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.206118 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.206242 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":254,"thread_start_us":83,"threads_started":1}
W20260808 06:02:23.206770 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
I20260808 06:02:23.207063 28384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:23.207718 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.207498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33722)
0808 06:02:23.207606 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.207670 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 06:02:23.207978 28330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
I20260808 06:02:23.208112 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:35733
--webserver_interface=127.13.229.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:23.208732 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.208532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33736)
0808 06:02:23.208634 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.208677 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:02:23.208892 28384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
I20260808 06:02:23.237954 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 1
I20260808 06:02:23.238782 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.238495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33742)
0808 06:02:23.238648 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.238710 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 06:02:23.239517 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.239307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33754)
0808 06:02:23.239413 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.239463 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:02:23.239683 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
I20260808 06:02:23.258160 28384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 1
I20260808 06:02:23.259001 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.258674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33762)
0808 06:02:23.258836 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.258911 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 06:02:23.259734 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.259511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33766)
0808 06:02:23.259617 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.259670 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 06:02:23.259881 28384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
I20260808 06:02:23.314875 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 2
I20260808 06:02:23.315883 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.315523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33778)
0808 06:02:23.315712 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.315788 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 06:02:23.316809 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.316588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33782)
0808 06:02:23.316695 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.316744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 06:02:23.316972 28305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
W20260808 06:02:23.322996 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:23.323257 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:23.323338 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:23.327436 28392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:23.327494 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:23.327514 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:23.327531 28392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:23.327574 28392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:23.328083 28384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 2
I20260808 06:02:23.328930 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.328591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33798)
0808 06:02:23.328763 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.328836 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 06:02:23.329653 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.329428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33812)
0808 06:02:23.329539 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.329591 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 06:02:23.329833 28384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35733: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35733: connect: Connection refused (error 111)
I20260808 06:02:23.333571 28392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45823,127.13.229.125:40437,127.13.229.124:35733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.229.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:23.334811 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:23.336004 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:23.344698 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:23.344693 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:23.344836 28392 server_base.cc:1034] running on GCE node
W20260808 06:02:23.345029 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:23.345628 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:23.346835 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:23.347980 28392 hybrid_clock.cc:648] HybridClock initialized: now 1786168943347962 us; error 27 us; skew 500 ppm
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:23.350874 28392 init.cc:377] Logged in from keytab as kudu/127.13.229.124@KRBTEST.COM (short username kudu)
I20260808 06:02:23.351760 28392 webserver.cc:460] Webserver started at http://127.13.229.124:33117/ using document root <none> and password file <none>
I20260808 06:02:23.352370 28392 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:23.352475 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:23.352706 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:23.354848 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data/instance:
uuid: "e4031033613b42758a15cf35178adff0"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "3c614ddaa9c3af59d7cf9781c63c7c4d"
server_key_iv: "cdf8ea732810543d46589d2837b7291b"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.355438 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal/instance:
uuid: "e4031033613b42758a15cf35178adff0"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "3c614ddaa9c3af59d7cf9781c63c7c4d"
server_key_iv: "cdf8ea732810543d46589d2837b7291b"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.359110 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:02:23.361609 28408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:23.363015 28392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:02:23.363189 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "e4031033613b42758a15cf35178adff0"
format_stamp: "Formatted at 2026-08-08 06:02:23 on dist-test-slave-zkpd"
server_key: "3c614ddaa9c3af59d7cf9781c63c7c4d"
server_key_iv: "cdf8ea732810543d46589d2837b7291b"
server_key_version: "encryptionkey@0"
I20260808 06:02:23.363324 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:23.375095 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:23.378110 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:23.378343 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:23.386155 28392 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.124:35733
I20260808 06:02:23.386430 28461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.124:35733 every 8 connection(s)
I20260808 06:02:23.387410 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
I20260808 06:02:23.390369 28462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:23.394266 28462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:23.395109 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28392
I20260808 06:02:23.395265 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786168783448266-14229-0/minicluster-data/master-2/wal/instance
I20260808 06:02:23.395535 14229 external_mini_cluster.cc:1428] Setting key 164b67f083e98573fde5bdabec165667
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:02:23.408219 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.396302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.13.229.126:45823)
0808 06:02:23.396462 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.396467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.397549 (+ 1082us) server_negotiation.cc:407] Connection header received
0808 06:02:23.398376 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.398378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.398423 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.398500 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.399513 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.400066 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.401158 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.401391 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.401475 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.404253 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.404257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.404260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.404297 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.405887 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.406416 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.406419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.406420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.406491 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.406863 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.406866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.406868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.407154 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.407228 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.407875 (+ 647us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.408039 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:02:23.409222 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.396769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:45823 (local address 127.0.0.1:60654)
0808 06:02:23.397399 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.397446 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.398161 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.398577 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.398582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.399017 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.399355 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.399378 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.400172 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.400175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.401034 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.401043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.401535 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.401538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.401704 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.402517 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.402533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.404077 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.405998 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.406005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.406015 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.406319 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.406582 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.406586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.406589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.406766 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.407254 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.407263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.407575 (+ 312us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.407788 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.408176 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":49,"spinlock_wait_cycles":4480,"thread_start_us":152,"threads_started":1}
I20260808 06:02:23.409222 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.396769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.13.229.124:35733)
0808 06:02:23.397286 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.397306 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.397348 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:02:23.398161 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.398191 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.398443 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.398630 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.399321 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.400382 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.401200 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.401628 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.401775 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.404708 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.404717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.404733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.404760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.406648 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.407110 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.407114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.407115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.407181 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.407461 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.407464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.407465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.407717 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.407938 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.408160 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.408293 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":353,"spinlock_wait_cycles":54400,"thread_start_us":170,"threads_started":1}
I20260808 06:02:23.410631 28462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:23.416149 28305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 3
I20260808 06:02:23.420771 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.411723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.13.229.126:45823)
0808 06:02:23.411836 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.411840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.411852 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:02:23.411887 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.411890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.411963 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.412040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.412715 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.413256 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.414055 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.414286 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.414385 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.416725 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.416729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.416733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.416759 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.418698 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419107 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.419110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.419112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.419204 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419515 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.419518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.419520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.419739 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.420010 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.420479 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.420616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:02:23.420928 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.411324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.13.229.125:40437)
0808 06:02:23.411468 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.411473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.411490 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:02:23.411571 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.411573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.411623 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.411711 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.412236 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.412992 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.413842 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.414190 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.414269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.416439 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.416446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.416452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.416483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.418362 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419471 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.419474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.419476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.419557 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419949 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.419953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.419956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.420178 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.420273 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.420658 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.420774 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:02:23.420984 28384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } attempt: 3
I20260808 06:02:23.421314 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.411192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:40437 (local address 127.0.0.1:37486)
0808 06:02:23.411391 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.411406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.411487 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.411744 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.411748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.411967 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.412106 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.412114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.413096 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.413099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.413720 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.413742 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.414294 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.414298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.414405 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.414858 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.414870 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.416069 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.419181 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.419184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.419366 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.419686 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.419689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.419691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.419781 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.420356 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.420360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.420470 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.420579 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.420719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:02:23.422178 28462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:02:23.425602 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.416542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33836)
0808 06:02:23.416827 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.416845 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.416922 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.417170 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.417176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.417491 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.417587 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.417593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.418555 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.418559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.419069 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.419077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.419637 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.419642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.419740 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.420264 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.420278 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.421928 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.424146 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.424152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.424154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.424407 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.424724 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.424727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.424729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.424814 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.425206 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.425211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.425314 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.425389 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.425493 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260808 06:02:23.426147 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.416803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.13.229.124:35733)
0808 06:02:23.416919 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.416925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.416952 (+ 27us) server_negotiation.cc:407] Connection header received
0808 06:02:23.416997 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.417000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.417032 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.417131 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.417702 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.418437 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.419159 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.419519 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.419619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.422092 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.422095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.422098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.422122 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.424030 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.424542 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.424545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.424547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.424624 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.424918 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.424921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.424923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.425097 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.425797 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.425869 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.425977 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:02:23.429982 28305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: Bootstrap starting.
Aug 08 06:02:23 dist-test-slave-zkpd krb5kdc[28221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:02:23.433444 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.421515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.13.229.124:35733)
0808 06:02:23.421852 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.421877 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.421891 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:02:23.421925 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.421928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.421971 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.422782 (+ 811us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.422890 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.424495 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.426214 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.426611 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.427394 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.429224 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.429230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.429234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.429258 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.431244 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.431774 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.431780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.431785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.431882 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432259 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.432262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.432265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.432513 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.433035 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.433110 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.433226 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":6528,"thread_start_us":109,"threads_started":1}
I20260808 06:02:23.433442 28305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:23.433898 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.421331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33846)
0808 06:02:23.421440 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.421455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.421529 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.422106 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.422111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.422280 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.422664 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.422673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.424606 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.424609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.425148 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.425157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.426737 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.426740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.426833 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.427504 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.427517 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.429069 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.431384 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.431387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.431389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.431638 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.432024 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.432029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.432132 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.432629 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.432633 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.432737 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.432830 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.433698 (+ 868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260808 06:02:23.433965 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.422815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35733 (local address 127.0.0.1:33848)
0808 06:02:23.422924 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:23.422939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:23.423004 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:23.423609 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:23.423613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:23.423874 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:23.424251 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.424257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.425287 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.425291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.426080 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:23.426092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.426804 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.426807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:23.426885 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.427856 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:23.427881 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:23.430097 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:23.432045 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.432049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.432352 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.432635 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:23.432640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:23.432716 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:23.433124 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:23.433127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:23.433201 (+ 74us) client_negotiation.cc:769] Sending connection context
0808 06:02:23.433295 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 06:02:23.433416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 06:02:23.434442 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.422963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.13.229.124:35733)
0808 06:02:23.423296 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.423307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.423320 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:02:23.423356 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.423359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.423400 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.423487 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.424366 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.425176 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.426202 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.426607 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.426712 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.430207 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.430210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.430212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.430235 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.431933 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432468 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.432471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.432473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.432546 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.432811 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.432816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.432819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.433012 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.433093 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.434143 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.434267 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260808 06:02:23.434675 28305 log.cc:824] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:23.437114 28305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: No bootstrap required, opened a new log
I20260808 06:02:23.437222 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.422584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.13.229.125:40437)
0808 06:02:23.422682 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.422686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.423437 (+ 751us) server_negotiation.cc:407] Connection header received
0808 06:02:23.423634 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.423639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.423688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.423739 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.424568 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.425928 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.427314 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.427629 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.427739 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.432831 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.432836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.432838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.432864 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.435009 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.435768 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.435772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.435773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.435860 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.436192 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.436195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.436197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.436396 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.436569 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.436964 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.437076 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":599,"server-negotiator.queue_time_us":29}
I20260808 06:02:23.437498 28462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: Bootstrap starting.
I20260808 06:02:23.437822 28384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: Bootstrap starting.
I20260808 06:02:23.440228 28462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:23.440361 28384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:23.440883 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:02:23.441143 28462 log.cc:824] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:23.441222 28384 log.cc:824] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:23.441991 28305 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.442315 28305 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:23.442411 28305 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb30655570b4fb587fd597389609d0c, State: Initialized, Role: FOLLOWER
I20260808 06:02:23.442984 28305 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.443293 28384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: No bootstrap required, opened a new log
I20260808 06:02:23.443315 28462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: No bootstrap required, opened a new log
I20260808 06:02:23.443667 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.443804 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.444633 28305 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: configured and running, proceeding with master startup.
W20260808 06:02:23.446633 28256 tablet.cc:2369] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:02:23.448392 28462 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.448647 28462 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:23.448702 28462 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4031033613b42758a15cf35178adff0, State: Initialized, Role: FOLLOWER
I20260808 06:02:23.448642 28384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.448843 28384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:23.448902 28384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311234c079c4445a8df9232581baaf38, State: Initialized, Role: FOLLOWER
I20260808 06:02:23.449278 28462 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.449333 28384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.450634 28462 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:23.450646 28384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:23.455963 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.445128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.13.229.124:35733)
0808 06:02:23.445237 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.445241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.445991 (+ 750us) server_negotiation.cc:407] Connection header received
0808 06:02:23.446162 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.446165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.446217 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.446265 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.447295 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.448148 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.449482 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.449848 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.449925 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.451841 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.451846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.451848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.451878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.453837 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.454350 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.454354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.454356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.454447 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.454789 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.454793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.454794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.455039 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.455175 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.455643 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.455771 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 06:02:23.456573 28514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:02:23.456647 28514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:02:23.456758 28477 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:23.456887 28492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.456923 28477 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.457013 28492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.450352 28234 master_runner.cc:186] Error getting master registration for 127.13.229.125:40437: OK, instance_id { permanent_uuid: "311234c079c4445a8df9232581baaf38" instance_seqno: 1786168943166341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:02:23.457777 28234 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:02:23.457897 28477 leader_election.cc:288] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311234c079c4445a8df9232581baaf38 (127.13.229.125:40437), e4031033613b42758a15cf35178adff0 (127.13.229.124:35733)
I20260808 06:02:23.464226 28435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb30655570b4fb587fd597389609d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4031033613b42758a15cf35178adff0" is_pre_election: true
I20260808 06:02:23.464633 28435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcb30655570b4fb587fd597389609d0c in term 0.
I20260808 06:02:23.465490 28251 leader_election.cc:302] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4031033613b42758a15cf35178adff0, fcb30655570b4fb587fd597389609d0c; no voters:
I20260808 06:02:23.466470 28477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:02:23.466571 28477 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:02:23.466625 28477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:23.467481 28392 master_runner.cc:263] No leader master found from master e4031033613b42758a15cf35178adff0
I20260808 06:02:23.467536 28392 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:02:23.468227 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.444651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.13.229.125:40437)
0808 06:02:23.444765 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.444769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.458779 (+ 14010us) server_negotiation.cc:407] Connection header received
0808 06:02:23.459044 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.459048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.459099 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.459153 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.460484 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.461204 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.462289 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.462599 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.462677 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.464364 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.464368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.464371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.464397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.466142 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.466667 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.466673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.466676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.466771 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.467171 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.467174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.467176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.467405 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.467526 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.467873 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.468062 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:02:23.468297 28477 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.468699 28477 leader_election.cc:288] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [CANDIDATE]: Term 1 election: Requested vote from peers 311234c079c4445a8df9232581baaf38 (127.13.229.125:40437), e4031033613b42758a15cf35178adff0 (127.13.229.124:35733)
I20260808 06:02:23.469506 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:23.449136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.13.229.126:45823)
0808 06:02:23.449239 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:23.449244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:23.449256 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:02:23.449290 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:23.449296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:23.449339 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:23.449420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:23.458809 (+ 9389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.459675 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.461069 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:23.461474 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:23.461561 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:23.463977 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:23.463982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:23.463984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:23.464017 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:23.466411 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.468055 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.468061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.468065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.468156 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:23.468489 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:23.468493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:23.468494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:23.468705 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:23.468819 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:23.469222 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 06:02:23.469341 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 06:02:23.470276 28356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb30655570b4fb587fd597389609d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311234c079c4445a8df9232581baaf38"
I20260808 06:02:23.470427 28435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb30655570b4fb587fd597389609d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4031033613b42758a15cf35178adff0"
I20260808 06:02:23.470480 28356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:23.470541 28435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:23.471951 28435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb30655570b4fb587fd597389609d0c in term 1.
I20260808 06:02:23.471959 28356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb30655570b4fb587fd597389609d0c in term 1.
I20260808 06:02:23.472263 28251 leader_election.cc:302] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4031033613b42758a15cf35178adff0, fcb30655570b4fb587fd597389609d0c; no voters:
W20260808 06:02:23.458451 28517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 06:02:23.459453 28357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb30655570b4fb587fd597389609d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311234c079c4445a8df9232581baaf38" is_pre_election: true
I20260808 06:02:23.472606 28477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:23.472693 28357 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fcb30655570b4fb587fd597389609d0c in term 1. Re-sending same reply.
I20260808 06:02:23.472836 28477 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [term 1 LEADER]: Becoming Leader. State: Replica: fcb30655570b4fb587fd597389609d0c, State: Running, Role: LEADER
I20260808 06:02:23.460668 28493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.473186 28493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.473191 28477 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } }
I20260808 06:02:23.475207 28313 master_runner.cc:263] No leader master found from master 311234c079c4445a8df9232581baaf38
I20260808 06:02:23.475275 28313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 06:02:23.475373 28517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:02:23.475797 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb30655570b4fb587fd597389609d0c. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.475960 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: This master's current role is: LEADER
W20260808 06:02:23.479542 28491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:02:23.479628 28491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:02:23.479743 28521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:23.480720 28521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:23.489343 28357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [term 1 FOLLOWER]: Refusing update from remote peer fcb30655570b4fb587fd597389609d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:23.489359 28435 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [term 1 FOLLOWER]: Refusing update from remote peer fcb30655570b4fb587fd597389609d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:23.490013 28477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:23.491484 28519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:23.494302 28492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb30655570b4fb587fd597389609d0c. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.494421 28492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.495319 28493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb30655570b4fb587fd597389609d0c. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.495441 28493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.503033 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.503181 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:23.503536 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.503652 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:23.504933 28493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.505051 28493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.505088 28492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcb30655570b4fb587fd597389609d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb30655570b4fb587fd597389609d0c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45823 } } peers { permanent_uuid: "311234c079c4445a8df9232581baaf38" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 40437 } } peers { permanent_uuid: "e4031033613b42758a15cf35178adff0" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35733 } } }
I20260808 06:02:23.505209 28492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:02:23.505497 28524 mvcc.cc:205] Tried to move back new op lower bound from 7316147992519254016 to 7316147992469442560. Current Snapshot: MvccSnapshot[applied={T|T < 7316147992519254016}]
I20260808 06:02:23.505911 28526 mvcc.cc:205] Tried to move back new op lower bound from 7316147992519254016 to 7316147992469442560. Current Snapshot: MvccSnapshot[applied={T|T < 7316147992519254016}]
I20260808 06:02:23.506112 28523 mvcc.cc:205] Tried to move back new op lower bound from 7316147992519254016 to 7316147992469442560. Current Snapshot: MvccSnapshot[applied={T|T < 7316147992519254016}]
I20260808 06:02:23.509799 28521 catalog_manager.cc:1347] Generated new cluster ID: 72644aeec5234ae49b5d7ab8614046fb
I20260808 06:02:23.509857 28521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:23.522778 28521 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:02:23.524428 28521 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:23.538949 28521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcb30655570b4fb587fd597389609d0c: Generated new TSK 0
I20260808 06:02:23.540038 28521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:02:24.458000 28514 catalog_manager.cc:1259] Loaded cluster ID: 72644aeec5234ae49b5d7ab8614046fb
I20260808 06:02:24.458091 28514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: loading cluster ID for follower catalog manager: success
I20260808 06:02:24.459839 28514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: acquiring CA information for follower catalog manager: success
I20260808 06:02:24.461054 28514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4031033613b42758a15cf35178adff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:02:24.476634 28517 catalog_manager.cc:1259] Loaded cluster ID: 72644aeec5234ae49b5d7ab8614046fb
I20260808 06:02:24.476701 28517 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: loading cluster ID for follower catalog manager: success
I20260808 06:02:24.478338 28517 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: acquiring CA information for follower catalog manager: success
I20260808 06:02:24.479488 28517 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 311234c079c4445a8df9232581baaf38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:02:24.609097 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28234
I20260808 06:02:24.617801 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28313
I20260808 06:02:24.625540 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28392
2026-08-08T06:02:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3056 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:02:24 dist-test-slave-zkpd krb5kdc[28540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:02:24 dist-test-slave-zkpd krb5kdc[28540](info): set up 2 sockets
Aug 08 06:02:24 dist-test-slave-zkpd krb5kdc[28540](info): commencing operation
krb5kdc: starting...
W20260808 06:02:25.693706 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:02:25 dist-test-slave-zkpd krb5kdc[28540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:02:25.814924 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:39111
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:25.926383 28555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:25.926644 28555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:25.926683 28555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:25.931146 28555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:25.931203 28555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:25.931223 28555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:25.931239 28555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:25.931254 28555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:25.937031 28555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:39111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:25.938011 28555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:25.938956 28555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:25.947408 28563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:25.947463 28560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:25.947408 28561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:25.947606 28555 server_base.cc:1034] running on GCE node
I20260808 06:02:25.948344 28555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:25.949419 28555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:25.950611 28555 hybrid_clock.cc:648] HybridClock initialized: now 1786168945950587 us; error 50 us; skew 500 ppm
Aug 08 06:02:25 dist-test-slave-zkpd krb5kdc[28540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168945, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:25.953366 28555 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:25.954270 28555 webserver.cc:460] Webserver started at http://127.13.229.126:41085/ using document root <none> and password file <none>
I20260808 06:02:25.954811 28555 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:25.954903 28555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:25.955150 28555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:25.957188 28555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "c4534546920e4ea9ae8e0b0992301f7e"
format_stamp: "Formatted at 2026-08-08 06:02:25 on dist-test-slave-zkpd"
server_key: "555b43c00b0eba9a2f77577ea3369b2f"
server_key_iv: "30b8c50e764056e9867fae5c71e9a797"
server_key_version: "encryptionkey@0"
I20260808 06:02:25.957749 28555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "c4534546920e4ea9ae8e0b0992301f7e"
format_stamp: "Formatted at 2026-08-08 06:02:25 on dist-test-slave-zkpd"
server_key: "555b43c00b0eba9a2f77577ea3369b2f"
server_key_iv: "30b8c50e764056e9867fae5c71e9a797"
server_key_version: "encryptionkey@0"
I20260808 06:02:25.961328 28555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:02:25.963572 28570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:25.964778 28555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:25.964926 28555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "c4534546920e4ea9ae8e0b0992301f7e"
format_stamp: "Formatted at 2026-08-08 06:02:25 on dist-test-slave-zkpd"
server_key: "555b43c00b0eba9a2f77577ea3369b2f"
server_key_iv: "30b8c50e764056e9867fae5c71e9a797"
server_key_version: "encryptionkey@0"
I20260808 06:02:25.965049 28555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:25.999963 28555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:26.002725 28555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:26.002945 28555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:26.010208 28555 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:39111
I20260808 06:02:26.010233 28623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:39111 every 8 connection(s)
I20260808 06:02:26.011145 28555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:26.013837 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:26.019284 28624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: Bootstrap starting.
I20260808 06:02:26.021092 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28555
I20260808 06:02:26.021224 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:02:26.021519 14229 external_mini_cluster.cc:1428] Setting key 7f7169ea212490b0055d7d54891cb105
I20260808 06:02:26.021994 28624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:26.022897 28624 log.cc:824] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:26.024868 28624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: No bootstrap required, opened a new log
I20260808 06:02:26.028566 28624 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } }
I20260808 06:02:26.028854 28624 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:26.028950 28624 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4534546920e4ea9ae8e0b0992301f7e, State: Initialized, Role: FOLLOWER
I20260808 06:02:26.029394 28624 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } }
I20260808 06:02:26.029551 28624 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:26.029637 28624 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:26.029757 28624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 0 FOLLOWER]: Advancing to term 1
Aug 08 06:02:26 dist-test-slave-zkpd krb5kdc[28540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:26.031018 28624 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } }
I20260808 06:02:26.031385 28624 leader_election.cc:302] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4534546920e4ea9ae8e0b0992301f7e; no voters:
I20260808 06:02:26.031741 28624 leader_election.cc:288] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:02:26.031821 28629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:26.032363 28629 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [term 1 LEADER]: Becoming Leader. State: Replica: c4534546920e4ea9ae8e0b0992301f7e, State: Running, Role: LEADER
I20260808 06:02:26.032759 28629 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } }
I20260808 06:02:26.033052 28624 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:26.034437 28630 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4534546920e4ea9ae8e0b0992301f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } } }
I20260808 06:02:26.034566 28630 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:26.034941 28631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4534546920e4ea9ae8e0b0992301f7e. Latest consensus state: current_term: 1 leader_uuid: "c4534546920e4ea9ae8e0b0992301f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4534546920e4ea9ae8e0b0992301f7e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 39111 } } }
I20260808 06:02:26.035105 28631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:26.037212 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:26.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.13.229.126:39111)
0808 06:02:26.023079 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:26.023096 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:26.023133 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:02:26.023994 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:26.024015 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:26.024326 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:26.024664 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:26.025520 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:26.026464 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:26.027340 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:26.027824 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:26.028040 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:26.031038 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:26.031043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:26.031057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:26.031084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:26.033307 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:26.033877 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:26.033885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:26.033895 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:26.034065 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:26.034504 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:26.034507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:26.034509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:26.035194 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:26.036098 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:26.036264 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 06:02:26.036515 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":136,"threads_started":1}
W20260808 06:02:26.038727 28645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:02:26.038836 28645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:02:26.038945 28638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:26.039727 28638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:26.040025 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:02:26.045348 28638 catalog_manager.cc:1347] Generated new cluster ID: 14b5e63168c34dfe9edb490a3319e4ed
I20260808 06:02:26.045423 28638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:26.061416 28638 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:02:26.062337 28638 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:26.066834 28638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4534546920e4ea9ae8e0b0992301f7e: Generated new TSK 0
I20260808 06:02:26.067468 28638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:02:26.095966 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28555
2026-08-08T06:02:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:02:26 dist-test-slave-zkpd krb5kdc[28651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:02:26 dist-test-slave-zkpd krb5kdc[28651](info): set up 2 sockets
Aug 08 06:02:26 dist-test-slave-zkpd krb5kdc[28651](info): commencing operation
krb5kdc: starting...
W20260808 06:02:27.149329 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:02:27.277005 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:27.389568 28665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:27.389840 28665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:27.389883 28665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:27.393791 28665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:27.393882 28665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:27.393922 28665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:27.393981 28665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:27.394021 28665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:27.399319 28665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:27.400435 28665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:27.401466 28665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:27.409973 28671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:27.409970 28670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:27.409976 28673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:27.410104 28665 server_base.cc:1034] running on GCE node
I20260808 06:02:27.410827 28665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:27.412009 28665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:27.413204 28665 hybrid_clock.cc:648] HybridClock initialized: now 1786168947413191 us; error 33 us; skew 500 ppm
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:27.416052 28665 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:27.416944 28665 webserver.cc:460] Webserver started at http://127.13.229.126:39939/ using document root <none> and password file <none>
I20260808 06:02:27.417516 28665 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:27.417609 28665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:27.417856 28665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:27.419978 28665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "74e446e208994a0d92083d053bc87e9f"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "2cfa1fd72fb4f8cc55d63a99e1bfcdca"
server_key_iv: "e4b41798e6695f68026fdedc91dcb4ba"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.420583 28665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "74e446e208994a0d92083d053bc87e9f"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "2cfa1fd72fb4f8cc55d63a99e1bfcdca"
server_key_iv: "e4b41798e6695f68026fdedc91dcb4ba"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.424261 28665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260808 06:02:27.426594 28680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.427790 28665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:27.427968 28665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "74e446e208994a0d92083d053bc87e9f"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "2cfa1fd72fb4f8cc55d63a99e1bfcdca"
server_key_iv: "e4b41798e6695f68026fdedc91dcb4ba"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.428082 28665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:27.450484 28665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:27.454201 28665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:27.454449 28665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:27.462776 28665 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38117
I20260808 06:02:27.463160 28735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38117 every 8 connection(s)
I20260808 06:02:27.464258 28665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:27.467537 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:27.473295 28736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap starting.
I20260808 06:02:27.473816 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28665
I20260808 06:02:27.473965 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:02:27.474253 14229 external_mini_cluster.cc:1428] Setting key 06d035fd059ed2e67ffc10b3cb95e7e0
I20260808 06:02:27.476270 28736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:27.477043 28736 log.cc:824] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:27.479241 28736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: No bootstrap required, opened a new log
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:27.483964 28736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:27.484283 28736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:27.484346 28736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Initialized, Role: FOLLOWER
I20260808 06:02:27.484946 28736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:27.485154 28736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:27.485234 28736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:27.485378 28736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:27.486666 28736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:27.487025 28736 leader_election.cc:302] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e446e208994a0d92083d053bc87e9f; no voters:
I20260808 06:02:27.487409 28736 leader_election.cc:288] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:02:27.487517 28741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:27.487739 28741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 LEADER]: Becoming Leader. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Running, Role: LEADER
I20260808 06:02:27.488178 28741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:27.488852 28736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:27.489848 28743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:27.489977 28743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:27.490204 28743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e446e208994a0d92083d053bc87e9f. Latest consensus state: current_term: 1 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:27.490280 28743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:27.490633 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:27.475654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.13.229.126:38117)
0808 06:02:27.476101 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:27.476123 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:27.476160 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:02:27.476841 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:27.476859 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:27.477258 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:27.477624 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:27.478503 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.479624 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.480835 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.481295 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.481438 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:27.484498 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:27.484504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:27.484522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:27.484555 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:27.486718 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.487181 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.487188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.487194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.487349 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.487696 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.487699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.487701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.488229 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:27.489068 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:27.489228 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 06:02:27.489543 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":157,"threads_started":1}
I20260808 06:02:27.491361 28750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:27.493933 28750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:27.499881 28750 catalog_manager.cc:1347] Generated new cluster ID: 76f38494a274438096ed921dc9bd8e92
I20260808 06:02:27.499967 28750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:27.519553 28750 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:02:27.521042 28750 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:27.527062 28750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Generated new TSK 0
I20260808 06:02:27.527747 28750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:02:27.565317 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:27.679957 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:27.680197 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:27.680274 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:27.684302 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:27.684365 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:27.684499 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:27.689777 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:27.690899 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:27.691994 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:27.701737 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:27.701828 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:27.701761 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:27.701920 28763 server_base.cc:1034] running on GCE node
I20260808 06:02:27.702387 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:27.703053 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:27.704236 28763 hybrid_clock.cc:648] HybridClock initialized: now 1786168947704212 us; error 49 us; skew 500 ppm
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:27.707046 28763 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:27.707930 28763 webserver.cc:460] Webserver started at http://127.13.229.65:44441/ using document root <none> and password file <none>
I20260808 06:02:27.708513 28763 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:27.708604 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:27.708845 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:27.710853 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "428cf9b04e904ee3aa5993eadd42c157"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "5a44bb378185b8585f41d9303db4e427"
server_key_iv: "62a7eb6d67646136610e9760d073fab7"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.711436 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "428cf9b04e904ee3aa5993eadd42c157"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "5a44bb378185b8585f41d9303db4e427"
server_key_iv: "62a7eb6d67646136610e9760d073fab7"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.715157 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:02:27.717521 28778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.718725 28763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:02:27.718884 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "428cf9b04e904ee3aa5993eadd42c157"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "5a44bb378185b8585f41d9303db4e427"
server_key_iv: "62a7eb6d67646136610e9760d073fab7"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.718998 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:27.728950 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:27.732148 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:27.732333 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:27.732916 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:27.733708 28763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:27.733793 28763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.733868 28763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:27.733912 28763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.742974 28763 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:34737
I20260808 06:02:27.743009 28897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:34737 every 8 connection(s)
I20260808 06:02:27.743857 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:27.752562 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28763
I20260808 06:02:27.752724 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:02:27.753064 14229 external_mini_cluster.cc:1428] Setting key 706e911dabaf9272756bf31a179ece0d
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:27.757876 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:27.745635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40631 (local address 127.13.229.126:38117)
0808 06:02:27.745785 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:27.745789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:27.746658 (+ 869us) server_negotiation.cc:407] Connection header received
0808 06:02:27.748003 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:27.748006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:27.748051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:27.748125 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:27.749261 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.749780 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.750849 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.751109 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.751188 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:27.754009 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:27.754012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:27.754014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:27.754037 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:27.755492 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.756147 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.756150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.756151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.756226 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.756661 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.756666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.756668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.756923 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:27.757042 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:27.757524 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 06:02:27.757671 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:02:27.758534 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:27.746062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:40631)
0808 06:02:27.746507 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:27.746547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:27.747759 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:27.748287 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:27.748294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:27.748654 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:27.749125 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:27.749138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.749887 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.749889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:27.750747 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:27.750755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.751241 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.751244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:27.751380 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:27.752019 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:27.752035 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:27.753842 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:27.755622 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:27.755631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:27.755648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:27.756051 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:27.756319 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:27.756324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:27.756329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:27.756532 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:27.757042 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:27.757047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:27.757269 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 06:02:27.757424 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 06:02:27.757674 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260808 06:02:27.759763 28898 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:27.760047 28898 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:27.760528 28898 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:27.762022 28698 ts_manager.cc:194] Registered new tserver with Master: 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65:34737)
I20260808 06:02:27.763132 28698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:40631
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:02:27.781299 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:27.897033 28903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:27.897276 28903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:27.897365 28903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:27.901836 28903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:27.901908 28903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:27.902009 28903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:27.907212 28903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:27.908320 28903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:27.909327 28903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:27.920163 28909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:27.920219 28903 server_base.cc:1034] running on GCE node
W20260808 06:02:27.920173 28908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:27.920408 28911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:27.920821 28903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:27.921506 28903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:27.922672 28903 hybrid_clock.cc:648] HybridClock initialized: now 1786168947922638 us; error 45 us; skew 500 ppm
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:27.930946 28903 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:27.932070 28903 webserver.cc:460] Webserver started at http://127.13.229.66:42501/ using document root <none> and password file <none>
I20260808 06:02:27.932698 28903 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:27.932763 28903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:27.932940 28903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:27.935024 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "2ed7a39f98614765bac58797f024e29c"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "772c78c1ce152eee3d056e1a04860557"
server_key_iv: "a04f943b6a09d8ad85dc3683632a021a"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.935539 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "2ed7a39f98614765bac58797f024e29c"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "772c78c1ce152eee3d056e1a04860557"
server_key_iv: "a04f943b6a09d8ad85dc3683632a021a"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.939265 28903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:02:27.941694 28919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.942947 28903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:27.943092 28903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "2ed7a39f98614765bac58797f024e29c"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "772c78c1ce152eee3d056e1a04860557"
server_key_iv: "a04f943b6a09d8ad85dc3683632a021a"
server_key_version: "encryptionkey@0"
I20260808 06:02:27.943203 28903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:27.961226 28903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:27.964179 28903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:27.964385 28903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:27.965009 28903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:27.965868 28903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:27.965947 28903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.966020 28903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:27.966073 28903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:27.975296 28903 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35603
I20260808 06:02:27.975338 29040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35603 every 8 connection(s)
I20260808 06:02:27.976257 28903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:27.978842 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 28903
I20260808 06:02:27.978994 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:02:27.979310 14229 external_mini_cluster.cc:1428] Setting key 5d0652ebe43f04c4172f44302eac2f7d
Aug 08 06:02:27 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:02:27.990971 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:27.978213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:34013 (local address 127.13.229.126:38117)
0808 06:02:27.978364 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:27.978370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:27.979269 (+ 899us) server_negotiation.cc:407] Connection header received
0808 06:02:27.980886 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:27.980890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:27.980942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:27.981021 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:27.982207 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.982727 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.983847 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.984119 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.984256 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:27.986740 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:27.986744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:27.986746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:27.986776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:27.988320 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.988867 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.988871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.988873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.988992 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:27.989446 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:27.989451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:27.989452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:27.989644 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:27.989762 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:27.990684 (+ 922us) server_negotiation.cc:299] Negotiation successful
0808 06:02:27.990793 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:27.991801 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:27.978621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:34013)
0808 06:02:27.979096 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:27.979148 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:27.980666 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:27.981195 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:27.981202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:27.981576 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:27.982054 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:27.982068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.982857 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.982867 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:27.983718 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:27.983728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:27.984300 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:27.984304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:27.984480 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:27.985114 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:27.985131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:27.986574 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:27.988460 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:27.988466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:27.988481 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:27.988736 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:27.989088 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:27.989092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:27.989094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:27.989357 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:27.989763 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:27.989770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:27.990124 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 06:02:27.990389 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 06:02:27.990701 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:02:27.993142 29041 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:27.993422 29041 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:27.993956 29041 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:27.995182 28698 ts_manager.cc:194] Registered new tserver with Master: 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603)
I20260808 06:02:27.995770 28698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:34013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:02:28.010264 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:28.125341 29047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:28.125626 29047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:28.125710 29047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:28.129755 29047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:28.129851 29047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:28.129956 29047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:28.135452 29047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:28.136588 29047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:28.137624 29047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:28.147454 29054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.147475 29056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.147629 29047 server_base.cc:1034] running on GCE node
W20260808 06:02:28.147876 29053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.148334 29047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:28.148972 29047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:28.150167 29047 hybrid_clock.cc:648] HybridClock initialized: now 1786168948150142 us; error 39 us; skew 500 ppm
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:28.153014 29047 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:28.153941 29047 webserver.cc:460] Webserver started at http://127.13.229.67:42185/ using document root <none> and password file <none>
I20260808 06:02:28.154513 29047 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:28.154604 29047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:28.154844 29047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:28.156980 29047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "bbc0df7ed8804d72966491202ec85158"
format_stamp: "Formatted at 2026-08-08 06:02:28 on dist-test-slave-zkpd"
server_key: "9d3574acd9a31e04a974ea2d897de3ab"
server_key_iv: "6169fe63276ac958969ff63f8447905d"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.157552 29047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "bbc0df7ed8804d72966491202ec85158"
format_stamp: "Formatted at 2026-08-08 06:02:28 on dist-test-slave-zkpd"
server_key: "9d3574acd9a31e04a974ea2d897de3ab"
server_key_iv: "6169fe63276ac958969ff63f8447905d"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.161341 29047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:02:28.163761 29064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.165046 29047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:28.165189 29047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "bbc0df7ed8804d72966491202ec85158"
format_stamp: "Formatted at 2026-08-08 06:02:28 on dist-test-slave-zkpd"
server_key: "9d3574acd9a31e04a974ea2d897de3ab"
server_key_iv: "6169fe63276ac958969ff63f8447905d"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.165305 29047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:28.186143 29047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:28.189519 29047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:28.189735 29047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:28.190384 29047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:28.191262 29047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:28.191332 29047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.191406 29047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:28.191455 29047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.200984 29047 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:46723
I20260808 06:02:28.201234 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:46723 every 8 connection(s)
I20260808 06:02:28.202155 29047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:02:28.207579 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29047
I20260808 06:02:28.207710 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:02:28.208038 14229 external_mini_cluster.cc:1428] Setting key b71f5e86f389342e835ec007a357c981
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:28.216053 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.204110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54197 (local address 127.13.229.126:38117)
0808 06:02:28.204262 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.204266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.205203 (+ 937us) server_negotiation.cc:407] Connection header received
0808 06:02:28.206641 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.206644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.206693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.206777 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:28.208364 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.208886 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.210098 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.210379 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.210479 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.212955 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:28.212958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:28.212961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:28.212984 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:28.214264 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.214712 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.214714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.214715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.214785 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.215089 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.215092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.215093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.215262 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:28.215331 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.215771 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.215862 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:28.216854 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.204547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:54197)
0808 06:02:28.205041 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.205085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.206333 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.206980 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.206987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.207476 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:28.208207 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.208230 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.209039 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.209043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.209973 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.209985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.210536 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.210539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.210708 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.211312 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:28.211328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:28.212802 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:28.214364 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:28.214369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:28.214380 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:28.214622 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:28.214863 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:28.214865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:28.214867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:28.215006 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:28.215345 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:28.215350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:28.215565 (+ 215us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.215711 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.216027 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260808 06:02:28.218030 29187 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:28.218292 29187 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:28.218796 29187 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:28.219820 28698 ts_manager.cc:194] Registered new tserver with Master: bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
I20260808 06:02:28.220547 28698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:54197
I20260808 06:02:28.221946 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:02:28.231076 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.224014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.13.229.126:38117)
0808 06:02:28.224164 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.224168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.224386 (+ 218us) server_negotiation.cc:407] Connection header received
0808 06:02:28.224479 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.224481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.224517 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.224579 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:28.225207 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.225681 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.226464 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.226663 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.226736 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.228133 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:28.228136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:28.228138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:28.228158 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:28.229437 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.229832 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.229835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.229836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.229916 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.230148 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.230151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.230152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.230362 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:28.230463 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.230767 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.230903 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 06:02:28.234259 28699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:02:28.237365 28699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:02:28.252286 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.245556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.13.229.67:46723)
0808 06:02:28.246631 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.246647 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.246671 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:02:28.246758 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.246762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.246971 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.247139 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.247606 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.248589 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.250558 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.251324 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.251424 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.251571 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.251982 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.252099 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":153,"threads_started":1}
I20260808 06:02:28.252300 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.246156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35603 (local address 127.0.0.1:37862)
0808 06:02:28.246881 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.246894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.246998 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.248077 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.248080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.248100 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.248187 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.248192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.249527 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.249530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.250302 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.250311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.251445 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.251448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.251711 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.251777 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.251929 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.252133 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":127,"thread_start_us":102,"threads_started":1}
I20260808 06:02:28.252820 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.246612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.13.229.66:35603)
0808 06:02:28.247491 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.247503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.247525 (+ 22us) server_negotiation.cc:407] Connection header received
0808 06:02:28.247602 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.247607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.247813 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.248003 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.248314 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.249369 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.250427 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.251299 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.251423 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.251571 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.252551 (+ 980us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.252651 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":121,"threads_started":1}
I20260808 06:02:28.253140 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.245345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:46723 (local address 127.0.0.1:36540)
0808 06:02:28.245726 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.245786 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.245928 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.247415 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.247417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.247430 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.247485 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.247489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.248714 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.248720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.250428 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.250440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.251445 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.251448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.251719 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.251778 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.251929 (+ 151us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.252133 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":335,"thread_start_us":94,"threads_started":1}
I20260808 06:02:28.254073 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.245682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34737 (local address 127.0.0.1:43936)
0808 06:02:28.246284 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.246300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.246387 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.247143 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.247148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.247167 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.247314 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.247319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.248756 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.248759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.250428 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.250440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.252840 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.252844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.253762 (+ 918us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.253789 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.253877 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.253951 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":149,"thread_start_us":126,"threads_started":1}
I20260808 06:02:28.254500 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.246171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.13.229.65:34737)
0808 06:02:28.246631 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.246647 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.246671 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:02:28.246752 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.246758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.246971 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.247139 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.247442 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.248598 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.251641 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.252708 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.252828 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.252990 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.254255 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.254353 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":109,"threads_started":1}
I20260808 06:02:28.255797 29115 tablet_service.cc:1467] Processing CreateTablet for tablet 62e8259cdd834a7ba1417752d6a55571 (DEFAULT_TABLE table=test-table [id=72350d6d0f504aa28ad381471516bf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:28.255829 28970 tablet_service.cc:1467] Processing CreateTablet for tablet 62e8259cdd834a7ba1417752d6a55571 (DEFAULT_TABLE table=test-table [id=72350d6d0f504aa28ad381471516bf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:28.256937 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e8259cdd834a7ba1417752d6a55571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:28.256955 28970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e8259cdd834a7ba1417752d6a55571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:28.257124 28830 tablet_service.cc:1467] Processing CreateTablet for tablet 62e8259cdd834a7ba1417752d6a55571 (DEFAULT_TABLE table=test-table [id=72350d6d0f504aa28ad381471516bf91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:28.258056 28830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e8259cdd834a7ba1417752d6a55571. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:28.263345 29204 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Bootstrap starting.
I20260808 06:02:28.264650 29205 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Bootstrap starting.
I20260808 06:02:28.264964 29206 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap starting.
I20260808 06:02:28.267071 29206 tablet_bootstrap.cc:654] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:28.267689 29204 tablet_bootstrap.cc:654] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:28.267689 29205 tablet_bootstrap.cc:654] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:28.268003 29206 log.cc:824] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:28.268636 29204 log.cc:824] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:28.268636 29205 log.cc:824] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:28.270318 29206 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: No bootstrap required, opened a new log
I20260808 06:02:28.270541 29206 ts_tablet_manager.cc:1397] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:02:28.270555 29205 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: No bootstrap required, opened a new log
I20260808 06:02:28.270754 29205 ts_tablet_manager.cc:1397] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 06:02:28.271379 29204 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: No bootstrap required, opened a new log
I20260808 06:02:28.271549 29204 ts_tablet_manager.cc:1397] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260808 06:02:28.274349 29206 raft_consensus.cc:348] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.274607 29206 raft_consensus.cc:374] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:28.274670 29206 raft_consensus.cc:729] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Initialized, Role: FOLLOWER
I20260808 06:02:28.274747 29205 raft_consensus.cc:348] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.275087 29205 raft_consensus.cc:374] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:28.275151 29205 raft_consensus.cc:729] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed7a39f98614765bac58797f024e29c, State: Initialized, Role: FOLLOWER
I20260808 06:02:28.275147 29206 consensus_queue.cc:260] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.275400 29204 raft_consensus.cc:348] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.275641 29204 raft_consensus.cc:374] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:28.275695 29204 raft_consensus.cc:729] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc0df7ed8804d72966491202ec85158, State: Initialized, Role: FOLLOWER
I20260808 06:02:28.275628 29205 consensus_queue.cc:260] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.276089 28898 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:28.276221 29204 consensus_queue.cc:260] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.276340 29206 ts_tablet_manager.cc:1428] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:02:28.276909 29187 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:28.277215 29204 ts_tablet_manager.cc:1428] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:02:28.277751 29041 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:28.277802 29205 ts_tablet_manager.cc:1428] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 06:02:28.348615 29211 raft_consensus.cc:482] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:28.348836 29211 raft_consensus.cc:504] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.349968 29211 leader_election.cc:288] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723), 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603)
I20260808 06:02:28.354064 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.350364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:55559 (local address 127.13.229.67:46723)
0808 06:02:28.350525 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.350529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.350761 (+ 232us) server_negotiation.cc:407] Connection header received
0808 06:02:28.350904 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.350907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.350978 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.351083 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.351400 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.351990 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353084 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.353485 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353618 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.353672 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.353869 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.353951 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:02:28.354096 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.350271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:46723 (local address 127.13.229.65:55559)
0808 06:02:28.350625 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.350639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.350797 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.351099 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.351102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.351122 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.351287 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.351292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.352105 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.352109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.352955 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.352964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.353613 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.353714 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.353758 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.353849 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.353919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"spinlock_wait_cycles":36224,"thread_start_us":150,"threads_started":1}
I20260808 06:02:28.354291 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.350884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:53705 (local address 127.13.229.66:35603)
0808 06:02:28.351024 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.351029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.351128 (+ 99us) server_negotiation.cc:407] Connection header received
0808 06:02:28.351298 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.351302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.351346 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.351444 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.351721 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.352301 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353314 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.353708 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353855 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.353922 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.354098 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.354169 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 06:02:28.354346 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.350706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35603 (local address 127.13.229.65:53705)
0808 06:02:28.351030 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.351045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.351187 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.351454 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.351457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.351472 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.351606 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.351612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.352423 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.352428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.353189 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.353197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.353849 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.353853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.353964 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.353993 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.354082 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.354208 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260808 06:02:28.354799 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e8259cdd834a7ba1417752d6a55571" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc0df7ed8804d72966491202ec85158" is_pre_election: true
I20260808 06:02:28.355155 29138 raft_consensus.cc:2454] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 0.
I20260808 06:02:28.355690 28780 leader_election.cc:302] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157, bbc0df7ed8804d72966491202ec85158; no voters:
I20260808 06:02:28.355859 28992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e8259cdd834a7ba1417752d6a55571" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed7a39f98614765bac58797f024e29c" is_pre_election: true
I20260808 06:02:28.355995 29211 raft_consensus.cc:2781] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:02:28.356086 29211 raft_consensus.cc:482] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:02:28.356141 29211 raft_consensus.cc:3037] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:28.356235 28992 raft_consensus.cc:2454] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 0.
I20260808 06:02:28.357285 29211 raft_consensus.cc:504] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.357583 29211 leader_election.cc:288] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 election: Requested vote from peers bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723), 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603)
I20260808 06:02:28.357939 29138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e8259cdd834a7ba1417752d6a55571" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc0df7ed8804d72966491202ec85158"
I20260808 06:02:28.358057 28992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e8259cdd834a7ba1417752d6a55571" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed7a39f98614765bac58797f024e29c"
I20260808 06:02:28.358081 29138 raft_consensus.cc:3037] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:28.358217 28992 raft_consensus.cc:3037] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:28.359149 29138 raft_consensus.cc:2454] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 1.
I20260808 06:02:28.359321 28992 raft_consensus.cc:2454] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 1.
I20260808 06:02:28.359444 28780 leader_election.cc:302] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157, bbc0df7ed8804d72966491202ec85158; no voters:
I20260808 06:02:28.359612 29211 raft_consensus.cc:2781] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:28.359849 29211 raft_consensus.cc:686] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 LEADER]: Becoming Leader. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Running, Role: LEADER
I20260808 06:02:28.360242 29211 consensus_queue.cc:237] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.363373 28699 catalog_manager.cc:5495] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "428cf9b04e904ee3aa5993eadd42c157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } health_report { overall_health: HEALTHY } } }
I20260808 06:02:28.377705 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.374023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.13.229.65:34737)
0808 06:02:28.374164 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.374170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.374184 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:02:28.374369 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.374371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.374426 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.374545 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:02:28.374783 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.375298 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.376236 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.376459 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.376570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.376876 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:02:28.377285 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:02:28.377428 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.377490 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.377545 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:02:28.383569 28992 raft_consensus.cc:1261] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Refusing update from remote peer 428cf9b04e904ee3aa5993eadd42c157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:28.383569 29138 raft_consensus.cc:1261] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Refusing update from remote peer 428cf9b04e904ee3aa5993eadd42c157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:28.384374 29211 consensus_queue.cc:1035] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:28.384591 29216 consensus_queue.cc:1035] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:28.402036 28699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:28.402223 28699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:28.404554 28699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Sending DeleteTablet for 3 replicas of tablet 62e8259cdd834a7ba1417752d6a55571
I20260808 06:02:28.405428 28830 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e8259cdd834a7ba1417752d6a55571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:28 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:43936
I20260808 06:02:28.405578 28970 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e8259cdd834a7ba1417752d6a55571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:28 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:37862
I20260808 06:02:28.405653 29115 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e8259cdd834a7ba1417752d6a55571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:28 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36540
I20260808 06:02:28.405993 29231 tablet_replica.cc:330] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: stopping tablet replica
I20260808 06:02:28.406148 29229 tablet_replica.cc:330] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: stopping tablet replica
I20260808 06:02:28.406174 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28763
I20260808 06:02:28.408195 29231 raft_consensus.cc:2229] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:28.408533 29231 raft_consensus.cc:2258] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:28.408737 29232 tablet_replica.cc:330] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: stopping tablet replica
I20260808 06:02:28.409025 29232 raft_consensus.cc:2229] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:28.409296 29232 raft_consensus.cc:2258] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:28.410162 29231 ts_tablet_manager.cc:1905] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:28.410657 29232 ts_tablet_manager.cc:1905] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:28.412523 29231 ts_tablet_manager.cc:1918] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:28.412636 29231 log.cc:1196] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/62e8259cdd834a7ba1417752d6a55571
I20260808 06:02:28.412967 29231 ts_tablet_manager.cc:1939] T 62e8259cdd834a7ba1417752d6a55571 P 2ed7a39f98614765bac58797f024e29c: Deleting consensus metadata
I20260808 06:02:28.413383 29232 ts_tablet_manager.cc:1918] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:28.413489 29232 log.cc:1196] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/62e8259cdd834a7ba1417752d6a55571
I20260808 06:02:28.413836 29232 ts_tablet_manager.cc:1939] T 62e8259cdd834a7ba1417752d6a55571 P bbc0df7ed8804d72966491202ec85158: Deleting consensus metadata
I20260808 06:02:28.414211 28683 catalog_manager.cc:4841] TS 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): tablet 62e8259cdd834a7ba1417752d6a55571 (table test-table [id=72350d6d0f504aa28ad381471516bf91]) successfully deleted
I20260808 06:02:28.414938 28683 catalog_manager.cc:4841] TS bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): tablet 62e8259cdd834a7ba1417752d6a55571 (table test-table [id=72350d6d0f504aa28ad381471516bf91]) successfully deleted
W20260808 06:02:28.416146 28684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:34737 (error 108)
I20260808 06:02:28.416188 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28903
I20260808 06:02:28.418905 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.418636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34737 (local address 127.0.0.1:43948)
0808 06:02:28.418774 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.418833 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 06:02:28.419148 28684 catalog_manager.cc:4584] TS 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65:34737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62e8259cdd834a7ba1417752d6a55571: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34737: connect: Connection refused (error 111)
I20260808 06:02:28.424494 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29047
I20260808 06:02:28.431258 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 28665
I20260808 06:02:28.439566 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--webserver_interface=127.13.229.126
--webserver_port=39939
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:28.552551 29234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:28.552827 29234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:28.552913 29234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:28.556870 29234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:28.556941 29234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:28.556980 29234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:28.557039 29234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:28.557073 29234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:28.562064 29234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:28.563028 29234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:28.564052 29234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:28.577332 29241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.577324 29243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.577438 29234 server_base.cc:1034] running on GCE node
W20260808 06:02:28.577363 29240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.578166 29234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:28.579293 29234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:28.580487 29234 hybrid_clock.cc:648] HybridClock initialized: now 1786168948580467 us; error 36 us; skew 500 ppm
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:28.583223 29234 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:28.584137 29234 webserver.cc:460] Webserver started at http://127.13.229.126:39939/ using document root <none> and password file <none>
I20260808 06:02:28.584709 29234 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:28.584805 29234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:28.588709 29234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 06:02:28.590848 29250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.592077 29234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:28.592216 29234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "74e446e208994a0d92083d053bc87e9f"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "2cfa1fd72fb4f8cc55d63a99e1bfcdca"
server_key_iv: "e4b41798e6695f68026fdedc91dcb4ba"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.592820 29234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:28.605350 29234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:28.608233 29234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:28.608450 29234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:28.615698 29234 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38117
I20260808 06:02:28.616003 29306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38117 every 8 connection(s)
I20260808 06:02:28.616922 29234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:28.621371 29307 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:02:28.623059 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap starting.
I20260808 06:02:28.626389 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29234
I20260808 06:02:28.626825 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:34737
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=44441
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:28.641052 29307 log.cc:824] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:28.649811 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:02:28.650200 29307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap complete.
I20260808 06:02:28.654541 29307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:28.654915 29307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Initialized, Role: FOLLOWER
I20260808 06:02:28.655448 29307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:28.655565 29307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:28.655658 29307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:28.655769 29307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:02:28.657676 29307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:28.658018 29307 leader_election.cc:302] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e446e208994a0d92083d053bc87e9f; no voters:
I20260808 06:02:28.658350 29307 leader_election.cc:288] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:02:28.658434 29311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:02:28.658644 29311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 LEADER]: Becoming Leader. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Running, Role: LEADER
I20260808 06:02:28.658949 29311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:28.659636 29307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:28.659870 29312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:28.660020 29312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:28.659960 29313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e446e208994a0d92083d053bc87e9f. Latest consensus state: current_term: 2 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:28.660069 29313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:28.660383 29315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:28.665244 29315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:28.665851 29315 catalog_manager.cc:1259] Loaded cluster ID: 76f38494a274438096ed921dc9bd8e92
I20260808 06:02:28.665953 29315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:28.667346 29315 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 06:02:28.672283 29315 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 06:02:28.677924 29315 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:28.678678 29315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Loaded TSK: 0
I20260808 06:02:28.679865 29315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:02:28.746472 29309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:28.746718 29309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:28.746760 29309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:28.750847 29309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:28.750913 29309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:28.750970 29309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:28.756260 29309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:34737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=44441
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:28.757239 29309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:28.758230 29309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:02:28.769366 29309 server_base.cc:1034] running on GCE node
W20260808 06:02:28.769299 29337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.769294 29335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.769572 29334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.770031 29309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:28.770704 29309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:28.771925 29309 hybrid_clock.cc:648] HybridClock initialized: now 1786168948771877 us; error 28 us; skew 500 ppm
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:28.774945 29309 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:28.775962 29309 webserver.cc:460] Webserver started at http://127.13.229.65:44441/ using document root <none> and password file <none>
I20260808 06:02:28.776520 29309 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:28.776592 29309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:28.780521 29309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:02:28.782809 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.784215 29309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:02:28.784368 29309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "428cf9b04e904ee3aa5993eadd42c157"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "5a44bb378185b8585f41d9303db4e427"
server_key_iv: "62a7eb6d67646136610e9760d073fab7"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.784953 29309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:28.804554 29309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:28.807760 29309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:28.807996 29309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:28.808606 29309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:28.809684 29352 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:02:28.813710 29309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:02:28.813798 29309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 06:02:28.813894 29309 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:02:28.815652 29309 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:02:28.815722 29309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 06:02:28.816082 29352 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap starting.
I20260808 06:02:28.826704 29309 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:34737
I20260808 06:02:28.826732 29465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:34737 every 8 connection(s)
I20260808 06:02:28.827641 29309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:28.834663 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29309
I20260808 06:02:28.835179 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:35603
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=42501
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:28.836911 29352 log.cc:824] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:28.844820 29352 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:02:28.845257 29352 tablet_bootstrap.cc:492] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap complete.
I20260808 06:02:28.846825 29352 ts_tablet_manager.cc:1397] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260808 06:02:28.846921 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.829852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:59655)
0808 06:02:28.830370 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.830424 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.831238 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.833496 (+ 2258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.833503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.833865 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:28.834320 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.834335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.835871 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.835873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.836780 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.836790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.837688 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.837693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.837923 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.838680 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:28.838704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:28.840364 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:28.843183 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:28.843189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:28.843201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:28.843546 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:28.843996 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:28.843998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:28.844000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:28.844165 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:28.845010 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:28.845018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:28.845326 (+ 308us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.845542 (+ 216us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.845889 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":153,"threads_started":1}
I20260808 06:02:28.847385 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.829900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:59655 (local address 127.13.229.126:38117)
0808 06:02:28.832176 (+ 2276us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.832194 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.832240 (+ 46us) server_negotiation.cc:407] Connection header received
0808 06:02:28.832537 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.832557 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.833050 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.833443 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:28.834517 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.835698 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.836915 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.837521 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.837698 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.840569 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:28.840578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:28.840597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:28.840635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:28.843046 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.843678 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.843687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.843694 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.843864 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:28.844278 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:28.844283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:28.844286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:28.844802 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:28.844991 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.845702 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.846056 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2120,"thread_start_us":154,"threads_started":1}
I20260808 06:02:28.849540 29352 raft_consensus.cc:348] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.850060 29352 raft_consensus.cc:729] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Initialized, Role: FOLLOWER
I20260808 06:02:28.850138 29466 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:28.850327 29466 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:28.850553 29352 consensus_queue.cc:260] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:28.850878 29466 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:28.851497 29352 ts_tablet_manager.cc:1428] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260808 06:02:28.853453 29269 ts_manager.cc:194] Registered new tserver with Master: 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65:34737)
I20260808 06:02:28.857262 29269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:59655
I20260808 06:02:28.859863 29466 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:28.863070 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.856817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.13.229.65:34737)
0808 06:02:28.858069 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:28.858085 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:28.858480 (+ 395us) server_negotiation.cc:407] Connection header received
0808 06:02:28.858697 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:28.858702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:28.859001 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:28.859164 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:28.859444 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.860564 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.861862 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.862387 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.862506 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.862634 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:28.862836 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 06:02:28.862927 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":268,"threads_started":1}
I20260808 06:02:28.863070 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:28.857574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34737 (local address 127.0.0.1:43962)
0808 06:02:28.858357 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:28.858401 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:28.858535 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:28.859174 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:28.859179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:28.859201 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:28.859323 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.859329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.860685 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.860688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.861722 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:28.861732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:28.862518 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:28.862522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:28.862630 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:28.862671 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 06:02:28.862795 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 06:02:28.862927 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":290,"threads_started":1}
I20260808 06:02:28.864143 29397 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e8259cdd834a7ba1417752d6a55571 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:28 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:43962
I20260808 06:02:28.864940 29475 tablet_replica.cc:330] stopping tablet replica
I20260808 06:02:28.865229 29475 raft_consensus.cc:2229] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:28.865458 29475 raft_consensus.cc:2258] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:28.866405 29475 ts_tablet_manager.cc:1905] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:28.870769 29475 ts_tablet_manager.cc:1918] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:28.870886 29475 log.cc:1196] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/62e8259cdd834a7ba1417752d6a55571
I20260808 06:02:28.871169 29475 ts_tablet_manager.cc:1939] T 62e8259cdd834a7ba1417752d6a55571 P 428cf9b04e904ee3aa5993eadd42c157: Deleting consensus metadata
I20260808 06:02:28.872983 29254 catalog_manager.cc:4841] TS 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65:34737): tablet 62e8259cdd834a7ba1417752d6a55571 (table test-table [id=72350d6d0f504aa28ad381471516bf91]) successfully deleted
W20260808 06:02:28.959725 29470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:28.959980 29470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:28.960021 29470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:28.964105 29470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:28.964157 29470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:28.964244 29470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:28.969358 29470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:35603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=42501
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:28.970393 29470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:28.971411 29470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:28.980846 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.980865 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:28.980845 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:28.981195 29470 server_base.cc:1034] running on GCE node
I20260808 06:02:28.981552 29470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:28.982151 29470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:28.983337 29470 hybrid_clock.cc:648] HybridClock initialized: now 1786168948983326 us; error 33 us; skew 500 ppm
Aug 08 06:02:28 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:28.985982 29470 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:28.986846 29470 webserver.cc:460] Webserver started at http://127.13.229.66:42501/ using document root <none> and password file <none>
I20260808 06:02:28.987403 29470 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:28.987506 29470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:28.991281 29470 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:28.993423 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:28.994728 29470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:02:28.994865 29470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "2ed7a39f98614765bac58797f024e29c"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "772c78c1ce152eee3d056e1a04860557"
server_key_iv: "a04f943b6a09d8ad85dc3683632a021a"
server_key_version: "encryptionkey@0"
I20260808 06:02:28.995407 29470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:29.021708 29470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:29.025053 29470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:29.025274 29470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:29.025923 29470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:29.026798 29470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:29.026878 29470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.026963 29470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:29.027015 29470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.036810 29470 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35603
I20260808 06:02:29.036830 29608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35603 every 8 connection(s)
I20260808 06:02:29.037901 29470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:29.044739 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29470
I20260808 06:02:29.045176 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:46723
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=42185
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:02:29 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:29.052078 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.039844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:33151 (local address 127.13.229.126:38117)
0808 06:02:29.040029 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.040032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.040902 (+ 870us) server_negotiation.cc:407] Connection header received
0808 06:02:29.042308 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.042311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.042385 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.042490 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:29.043670 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.044293 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.045566 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.045807 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.045926 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.048424 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:29.048427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:29.048429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:29.048453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:29.049915 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.050429 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.050433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.050435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.050519 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.050864 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.050869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.050874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.051053 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:29.051207 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.051699 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.051849 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 06:02:29.052848 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.040286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:33151)
0808 06:02:29.040754 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.040798 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.042090 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.042674 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.042681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.043047 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:29.043521 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.043535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.044430 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.044440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.045444 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.045456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.045968 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.045972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.046154 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.046774 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:29.046791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:29.048245 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:29.050036 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:29.050042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:29.050057 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:29.050321 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:29.050615 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:29.050619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:29.050621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:29.050766 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:29.051185 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:29.051191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:29.051445 (+ 254us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.051638 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.052027 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260808 06:02:29.054250 29609 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:29.054502 29609 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:29.054998 29609 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:29.056175 29269 ts_manager.cc:194] Registered new tserver with Master: 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603)
I20260808 06:02:29.057073 29269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:33151
W20260808 06:02:29.163074 29612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:29.163328 29612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:29.163372 29612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:29.167999 29612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:29.168059 29612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:29.168116 29612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:29.173452 29612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:46723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=42185
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:29.174463 29612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:29.175438 29612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:29.185178 29617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:29.185251 29620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:29.185199 29618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:29.185456 29612 server_base.cc:1034] running on GCE node
I20260808 06:02:29.186733 29612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:29.187328 29612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:29.188489 29612 hybrid_clock.cc:648] HybridClock initialized: now 1786168949188464 us; error 39 us; skew 500 ppm
Aug 08 06:02:29 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:29.191166 29612 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:29.192096 29612 webserver.cc:460] Webserver started at http://127.13.229.67:42185/ using document root <none> and password file <none>
I20260808 06:02:29.192659 29612 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:29.192751 29612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:29.196563 29612 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:29.198688 29628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.200106 29612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:29.200227 29612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "bbc0df7ed8804d72966491202ec85158"
format_stamp: "Formatted at 2026-08-08 06:02:28 on dist-test-slave-zkpd"
server_key: "9d3574acd9a31e04a974ea2d897de3ab"
server_key_iv: "6169fe63276ac958969ff63f8447905d"
server_key_version: "encryptionkey@0"
I20260808 06:02:29.200740 29612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:29.222278 29612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:29.225106 29612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:29.225302 29612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:29.225898 29612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:29.226910 29612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:29.226964 29612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.227036 29612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:29.227077 29612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.236465 29612 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:46723
I20260808 06:02:29.236493 29746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:46723 every 8 connection(s)
I20260808 06:02:29.237479 29612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:02:29.243458 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29612
Aug 08 06:02:29 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:29.251744 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.244202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.13.229.126:38117)
0808 06:02:29.244463 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.244467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.244529 (+ 62us) server_negotiation.cc:407] Connection header received
0808 06:02:29.244655 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.244658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.244707 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.244794 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:29.245509 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.246034 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.246883 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.247117 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.247222 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.248627 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:29.248630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:29.248633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:29.248675 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:29.250112 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250538 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.250543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.250544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.250626 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250893 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.250896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.250898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.251076 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:29.251298 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.251429 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.251545 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260808 06:02:29.252056 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.239343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:36073 (local address 127.13.229.126:38117)
0808 06:02:29.239480 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.239484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.240591 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 06:02:29.241947 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.241950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.242000 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.242080 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:29.243209 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.243803 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.245362 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.245606 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.245720 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.248349 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:29.248353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:29.248356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:29.248381 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:29.250013 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250513 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.250517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.250519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.250600 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250973 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:29.250978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:29.250979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:29.251149 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:29.251250 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.251809 (+ 559us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.251946 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:29.253185 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.239718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:36073)
0808 06:02:29.240439 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.240483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.241727 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.242246 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.242253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.242598 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:29.243082 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.243096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.243994 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.243997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.245229 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.245239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.245759 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.245763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.245945 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.246566 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:29.246583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:29.248188 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:29.250132 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:29.250153 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:29.250407 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:29.250720 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:29.250724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:29.250725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:29.250867 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:29.251268 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:29.251275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:29.251505 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.251747 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.252144 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":167,"threads_started":1}
I20260808 06:02:29.254328 29747 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38117
I20260808 06:02:29.254527 29747 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:29.254951 29747 heartbeater.cc:511] Master 127.13.229.126:38117 requested a full tablet report, sending...
I20260808 06:02:29.255841 29269 ts_manager.cc:194] Registered new tserver with Master: bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
I20260808 06:02:29.256407 29269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:36073
I20260808 06:02:29.265762 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:02:29.269877 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.266400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.13.229.126:38117)
0808 06:02:29.266518 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.266521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.266742 (+ 221us) server_negotiation.cc:407] Connection header received
0808 06:02:29.266875 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.266878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.266973 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.267051 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:02:29.267263 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.267751 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.268623 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.268828 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.268905 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.269175 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:02:29.269583 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:02:29.269671 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.269722 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.269765 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:02:29.270594 29269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:02:29.274178 29269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:02:29.282505 29397 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 (DEFAULT_TABLE table=test-table [id=34409cdb32af48b194cabb0b2e1d3b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:29.283187 29397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8f75f7e68f4708b2459c4823dfa3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:29.285820 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.281166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:46723 (local address 127.0.0.1:36546)
0808 06:02:29.281504 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.281517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.281600 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.282372 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.282375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.282392 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:29.282476 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.282483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.283788 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.283791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.284614 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.284623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.285429 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.285440 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.285523 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.285546 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.285605 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.285677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260808 06:02:29.286233 29352 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap starting.
I20260808 06:02:29.286473 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.280954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35603 (local address 127.0.0.1:37870)
0808 06:02:29.281067 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.281083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.281162 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.282179 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.282182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.282202 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:29.282311 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.282318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.283837 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.283841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.284941 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.284953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.286098 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.286102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.286187 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.286213 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.286274 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.286348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 06:02:29.286554 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.281154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.13.229.66:35603)
0808 06:02:29.281629 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.281638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.281659 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:02:29.281758 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.281764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.282009 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.282176 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:29.282586 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.283673 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.285110 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.285952 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.286093 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.286251 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.286325 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.286415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":150,"threads_started":1}
I20260808 06:02:29.286994 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.281503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.13.229.67:46723)
0808 06:02:29.281934 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.281945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.281965 (+ 20us) server_negotiation.cc:407] Connection header received
0808 06:02:29.282038 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.282042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.282226 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.282361 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:29.282604 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.283635 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.284740 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.285304 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.285430 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.285583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.285649 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.285736 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":205,"threads_started":1}
I20260808 06:02:29.287185 29352 tablet_bootstrap.cc:654] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:29.288604 29352 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: No bootstrap required, opened a new log
I20260808 06:02:29.288699 29352 ts_tablet_manager.cc:1397] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 06:02:29.289115 29352 raft_consensus.cc:348] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.289260 29352 raft_consensus.cc:374] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:29.289309 29352 raft_consensus.cc:729] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Initialized, Role: FOLLOWER
I20260808 06:02:29.289422 29352 consensus_queue.cc:260] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.289680 29352 ts_tablet_manager.cc:1428] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:02:29.289686 29540 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 (DEFAULT_TABLE table=test-table [id=34409cdb32af48b194cabb0b2e1d3b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:29.290072 29677 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 (DEFAULT_TABLE table=test-table [id=34409cdb32af48b194cabb0b2e1d3b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:29.291275 29677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8f75f7e68f4708b2459c4823dfa3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:29.291927 29540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8f75f7e68f4708b2459c4823dfa3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:29.299808 29760 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Bootstrap starting.
I20260808 06:02:29.300210 29761 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Bootstrap starting.
I20260808 06:02:29.302980 29760 tablet_bootstrap.cc:654] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:29.302980 29761 tablet_bootstrap.cc:654] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:29.303944 29760 log.cc:824] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:29.303944 29761 log.cc:824] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:29.306192 29761 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: No bootstrap required, opened a new log
I20260808 06:02:29.306192 29760 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: No bootstrap required, opened a new log
I20260808 06:02:29.306393 29760 ts_tablet_manager.cc:1397] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260808 06:02:29.306393 29761 ts_tablet_manager.cc:1397] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260808 06:02:29.310416 29761 raft_consensus.cc:348] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.310410 29760 raft_consensus.cc:348] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.310688 29760 raft_consensus.cc:374] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:29.310688 29761 raft_consensus.cc:374] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:29.310750 29761 raft_consensus.cc:729] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc0df7ed8804d72966491202ec85158, State: Initialized, Role: FOLLOWER
I20260808 06:02:29.310750 29760 raft_consensus.cc:729] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed7a39f98614765bac58797f024e29c, State: Initialized, Role: FOLLOWER
I20260808 06:02:29.311229 29761 consensus_queue.cc:260] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.311229 29760 consensus_queue.cc:260] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.312064 29747 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:29.312291 29761 ts_tablet_manager.cc:1428] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 06:02:29.312291 29760 ts_tablet_manager.cc:1428] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 06:02:29.312417 29609 heartbeater.cc:503] Master 127.13.229.126:38117 was elected leader, sending a full tablet report...
I20260808 06:02:29.327657 29472 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:29.327877 29472 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.329034 29472 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
W20260808 06:02:29.329988 29467 tablet.cc:2369] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:02:29.333441 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.329302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35603 (local address 127.13.229.65:42629)
0808 06:02:29.329472 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.329489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.329595 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.330026 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.330030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.330072 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:29.330211 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.330219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.331023 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.331027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.332165 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.332175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.332965 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.332969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.333077 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.333104 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.333188 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.333280 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":2432}
I20260808 06:02:29.333455 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.329387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:42629 (local address 127.13.229.66:35603)
0808 06:02:29.329535 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.329540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.329555 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:02:29.329737 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.329743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.329803 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.329889 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:29.330324 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.330905 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.332292 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.332852 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.332946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.332987 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.333211 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.333312 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:29.333801 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.329748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:33735 (local address 127.13.229.67:46723)
0808 06:02:29.329957 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.329962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.330029 (+ 67us) server_negotiation.cc:407] Connection header received
0808 06:02:29.330238 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.330244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.330305 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.330403 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:29.330695 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.331330 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.332683 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.333195 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.333301 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.333347 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.333590 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.333673 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260808 06:02:29.334131 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.329626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:46723 (local address 127.13.229.65:33735)
0808 06:02:29.329947 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.329960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.330101 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.330442 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:29.330445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:29.330463 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:29.330584 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.330589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.331451 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.331454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.332558 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:29.332569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.333328 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.333332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:29.333429 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.333452 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:02:29.333947 (+ 495us) client_negotiation.cc:240] Negotiation successful
0808 06:02:29.334027 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260808 06:02:29.334211 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8f75f7e68f4708b2459c4823dfa3a0" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed7a39f98614765bac58797f024e29c" is_pre_election: true
I20260808 06:02:29.334601 29561 raft_consensus.cc:2454] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 0.
I20260808 06:02:29.334753 29698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8f75f7e68f4708b2459c4823dfa3a0" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc0df7ed8804d72966491202ec85158" is_pre_election: true
I20260808 06:02:29.335064 29698 raft_consensus.cc:2454] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 0.
I20260808 06:02:29.335114 29346 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed7a39f98614765bac58797f024e29c, 428cf9b04e904ee3aa5993eadd42c157; no voters:
I20260808 06:02:29.335433 29472 raft_consensus.cc:2781] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:02:29.335523 29472 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:02:29.335569 29472 raft_consensus.cc:3037] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:29.336722 29472 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.337054 29472 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 election: Requested vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
I20260808 06:02:29.337376 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8f75f7e68f4708b2459c4823dfa3a0" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ed7a39f98614765bac58797f024e29c"
I20260808 06:02:29.337448 29698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8f75f7e68f4708b2459c4823dfa3a0" candidate_uuid: "428cf9b04e904ee3aa5993eadd42c157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc0df7ed8804d72966491202ec85158"
I20260808 06:02:29.337517 29561 raft_consensus.cc:3037] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:29.337586 29698 raft_consensus.cc:3037] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:29.338580 29561 raft_consensus.cc:2454] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 1.
I20260808 06:02:29.338624 29698 raft_consensus.cc:2454] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428cf9b04e904ee3aa5993eadd42c157 in term 1.
I20260808 06:02:29.338871 29346 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ed7a39f98614765bac58797f024e29c, 428cf9b04e904ee3aa5993eadd42c157; no voters:
I20260808 06:02:29.339054 29472 raft_consensus.cc:2781] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:29.339300 29472 raft_consensus.cc:686] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 LEADER]: Becoming Leader. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Running, Role: LEADER
I20260808 06:02:29.339564 29472 consensus_queue.cc:237] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.342382 29268 catalog_manager.cc:5495] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "428cf9b04e904ee3aa5993eadd42c157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } health_report { overall_health: HEALTHY } } }
I20260808 06:02:29.368130 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.364551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.13.229.65:34737)
0808 06:02:29.364791 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:29.364795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:29.364808 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:02:29.364878 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:29.364880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:29.364933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:29.365067 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:02:29.365315 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.365827 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.366666 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:29.366892 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:29.367003 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:29.367286 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:02:29.367709 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:02:29.367839 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:29.367911 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 06:02:29.367972 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260808 06:02:29.372135 29698 raft_consensus.cc:1261] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Refusing update from remote peer 428cf9b04e904ee3aa5993eadd42c157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:29.372148 29561 raft_consensus.cc:1261] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Refusing update from remote peer 428cf9b04e904ee3aa5993eadd42c157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:29.372867 29472 consensus_queue.cc:1035] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:29.373041 29768 consensus_queue.cc:1035] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:29.381460 29774 mvcc.cc:205] Tried to move back new op lower bound from 7316148016620449792 to 7316148016496369664. Current Snapshot: MvccSnapshot[applied={T|T < 7316148016620449792}]
I20260808 06:02:29.385440 29775 mvcc.cc:205] Tried to move back new op lower bound from 7316148016620449792 to 7316148016496369664. Current Snapshot: MvccSnapshot[applied={T|T < 7316148016620449792 or (T in {7316148016620449792})}]
I20260808 06:02:29.388751 29268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:29.388933 29268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:29.391340 29268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Sending DeleteTablet for 3 replicas of tablet 2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:29.392143 29540 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:29 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:37870
I20260808 06:02:29.392285 29397 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:29 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:43962
I20260808 06:02:29.392639 29784 tablet_replica.cc:330] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: stopping tablet replica
I20260808 06:02:29.392673 29677 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8f75f7e68f4708b2459c4823dfa3a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:29 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:36546
I20260808 06:02:29.392967 29784 raft_consensus.cc:2229] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:29.393275 29784 raft_consensus.cc:2258] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:29.393353 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29309
I20260808 06:02:29.395829 29784 ts_tablet_manager.cc:1905] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:29.393452 29785 tablet_replica.cc:330] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: stopping tablet replica
I20260808 06:02:29.397487 29785 raft_consensus.cc:2229] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:29.397812 29785 raft_consensus.cc:2258] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:29.399118 29784 ts_tablet_manager.cc:1918] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:29.399222 29784 log.cc:1196] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:29.399336 29785 ts_tablet_manager.cc:1905] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:29.399601 29784 ts_tablet_manager.cc:1939] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 2ed7a39f98614765bac58797f024e29c: Deleting consensus metadata
I20260808 06:02:29.400894 29253 catalog_manager.cc:4841] TS 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): tablet 2d8f75f7e68f4708b2459c4823dfa3a0 (table test-table [id=34409cdb32af48b194cabb0b2e1d3b2f]) successfully deleted
I20260808 06:02:29.401894 29785 ts_tablet_manager.cc:1918] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:29.402017 29785 log.cc:1196] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:29.402376 29785 ts_tablet_manager.cc:1939] T 2d8f75f7e68f4708b2459c4823dfa3a0 P bbc0df7ed8804d72966491202ec85158: Deleting consensus metadata
W20260808 06:02:29.403093 29254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:34737 (error 108)
I20260808 06:02:29.403528 29253 catalog_manager.cc:4841] TS bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): tablet 2d8f75f7e68f4708b2459c4823dfa3a0 (table test-table [id=34409cdb32af48b194cabb0b2e1d3b2f]) successfully deleted
I20260808 06:02:29.404162 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.403825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:34737 (local address 127.0.0.1:43970)
0808 06:02:29.404005 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.404091 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 06:02:29.404165 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29470
W20260808 06:02:29.405320 29254 catalog_manager.cc:4584] TS 428cf9b04e904ee3aa5993eadd42c157 (127.13.229.65:34737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d8f75f7e68f4708b2459c4823dfa3a0: Network error: Client connection negotiation failed: client connection to 127.13.229.65:34737: connect: Connection refused (error 111)
I20260808 06:02:29.413020 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29612
I20260808 06:02:29.419799 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29234
I20260808 06:02:29.427225 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--webserver_interface=127.13.229.126
--webserver_port=39939
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:29.540836 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:29.541139 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:29.541220 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:29.545140 29787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:29.545226 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:29.545264 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:29.545322 29787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:29.545361 29787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:29.550462 29787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:29.551497 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:29.552533 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:29.560928 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:29.560948 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:29.561116 29787 server_base.cc:1034] running on GCE node
W20260808 06:02:29.560947 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:29.562069 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:29.563151 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:29.564334 29787 hybrid_clock.cc:648] HybridClock initialized: now 1786168949564297 us; error 49 us; skew 500 ppm
Aug 08 06:02:29 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:29.567070 29787 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:29.567878 29787 webserver.cc:460] Webserver started at http://127.13.229.126:39939/ using document root <none> and password file <none>
I20260808 06:02:29.568447 29787 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:29.568507 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:29.572167 29787 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 06:02:29.574184 29803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.575392 29787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:29.575490 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "74e446e208994a0d92083d053bc87e9f"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "2cfa1fd72fb4f8cc55d63a99e1bfcdca"
server_key_iv: "e4b41798e6695f68026fdedc91dcb4ba"
server_key_version: "encryptionkey@0"
I20260808 06:02:29.576071 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:29.592702 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:29.595994 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:29.596148 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:29.603017 29787 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38117
I20260808 06:02:29.603304 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38117 every 8 connection(s)
I20260808 06:02:29.604147 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:29.608338 29858 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:02:29.609905 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap starting.
I20260808 06:02:29.613716 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29787
I20260808 06:02:29.614243 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:34737
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=44441
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:29.629505 29858 log.cc:824] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:29.640789 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:02:29.641175 29858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f: Bootstrap complete.
I20260808 06:02:29.645360 29858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:29.645758 29858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Initialized, Role: FOLLOWER
I20260808 06:02:29.646262 29858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:29.646377 29858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:29.646476 29858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:29.646586 29858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 2 FOLLOWER]: Advancing to term 3
I20260808 06:02:29.648521 29858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:29.648854 29858 leader_election.cc:302] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e446e208994a0d92083d053bc87e9f; no voters:
I20260808 06:02:29.649184 29858 leader_election.cc:288] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 06:02:29.649355 29862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 3 FOLLOWER]: Leader election won for term 3
I20260808 06:02:29.649590 29862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [term 3 LEADER]: Becoming Leader. State: Replica: 74e446e208994a0d92083d053bc87e9f, State: Running, Role: LEADER
I20260808 06:02:29.649881 29862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } }
I20260808 06:02:29.650426 29858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:29.651255 29863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:29.651382 29863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:29.651770 29864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e446e208994a0d92083d053bc87e9f. Latest consensus state: current_term: 3 leader_uuid: "74e446e208994a0d92083d053bc87e9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e446e208994a0d92083d053bc87e9f" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38117 } } }
I20260808 06:02:29.651966 29864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e446e208994a0d92083d053bc87e9f [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:29.652266 29872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:29.657508 29872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:29.658401 29872 catalog_manager.cc:1259] Loaded cluster ID: 76f38494a274438096ed921dc9bd8e92
I20260808 06:02:29.658464 29872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 06:02:29.659467 29872 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08343e99ad google::LogMessage::Fail() at ??:0
@ 0x7f08343ec88a google::LogMessage::SendToLog() at ??:0
@ 0x7f08343e94de google::LogMessage::Flush() at ??:0
@ 0x7f08343eaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f083ed28fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f083ed29d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f083ed26579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f083ed6a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f083663b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f083663bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f083663d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f083662c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0837c706db start_thread at ??:0
@ 0x7f0832ee761f clone at ??:0
W20260808 06:02:29.764808 29860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:29.765168 29860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:29.765245 29860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:29.771355 29860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:29.771451 29860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:29.771540 29860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:29.779642 29860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:34737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=44441
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:29.781131 29860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:29.782480 29860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:29.794325 29900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:29.794400 29860 server_base.cc:1034] running on GCE node
W20260808 06:02:29.794553 29903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:29.794332 29901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:29.795073 29860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:29.795876 29860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:29.797107 29860 hybrid_clock.cc:648] HybridClock initialized: now 1786168949797072 us; error 40 us; skew 500 ppm
Aug 08 06:02:29 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:29.800647 29860 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:29.801785 29860 webserver.cc:460] Webserver started at http://127.13.229.65:44441/ using document root <none> and password file <none>
I20260808 06:02:29.802515 29860 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:29.802611 29860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:29.807747 29860 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:02:29.810421 29910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:29.811947 29860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:02:29.812077 29860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "428cf9b04e904ee3aa5993eadd42c157"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "5a44bb378185b8585f41d9303db4e427"
server_key_iv: "62a7eb6d67646136610e9760d073fab7"
server_key_version: "encryptionkey@0"
I20260808 06:02:29.812750 29860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:29.830205 29860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:29.833842 29860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:29.834051 29860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:29.834801 29860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:29.836166 29918 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:02:29.840266 29860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:02:29.840341 29860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 06:02:29.840408 29860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:02:29.842649 29860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:02:29.842710 29860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 06:02:29.843032 29918 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap starting.
I20260808 06:02:29.853847 29860 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:34737
I20260808 06:02:29.854173 30032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:34737 every 8 connection(s)
I20260808 06:02:29.855048 29860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:29.856283 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 29860
I20260808 06:02:29.856916 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:35603
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=42501
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:29.865746 29918 log.cc:824] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:29.875166 29918 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:02:29.875622 29918 tablet_bootstrap.cc:492] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Bootstrap complete.
I20260808 06:02:29.877353 29918 ts_tablet_manager.cc:1397] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.015s
I20260808 06:02:29.880770 29918 raft_consensus.cc:348] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.881349 29918 raft_consensus.cc:729] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428cf9b04e904ee3aa5993eadd42c157, State: Initialized, Role: FOLLOWER
I20260808 06:02:29.881922 29918 consensus_queue.cc:260] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:29.883004 29918 ts_tablet_manager.cc:1428] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260808 06:02:29.916540 30036 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:02:29.860480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:50361)
0808 06:02:29.861108 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.861178 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:29.862429 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:29.915643 (+ 53214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":182,"threads_started":1}
W20260808 06:02:29.917114 30033 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38117 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38117: Client connection negotiation failed: client connection to 127.13.229.126:38117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:02:29.917918 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.917629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:44003)
0808 06:02:29.917784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.917859 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 06:02:29.918659 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:29.918405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:32819)
0808 06:02:29.918549 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:29.918609 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 06:02:29.918888 30033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:02:29.981629 30035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:29.981891 30035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:29.981932 30035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:29.985922 30035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:29.985977 30035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:29.986033 30035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:29.991055 30035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:35603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=42501
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:29.992079 30035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:29.993049 30035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:30.002708 30043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:30.002699 30044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:30.002735 30046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:30.002993 30035 server_base.cc:1034] running on GCE node
I20260808 06:02:30.003510 30035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:30.004207 30035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:30.005407 30035 hybrid_clock.cc:648] HybridClock initialized: now 1786168950005376 us; error 40 us; skew 500 ppm
Aug 08 06:02:30 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168950, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:30.008212 30035 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:30.009083 30035 webserver.cc:460] Webserver started at http://127.13.229.66:42501/ using document root <none> and password file <none>
I20260808 06:02:30.009711 30035 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:30.009829 30035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:30.013721 30035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:02:30.015794 30053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.017063 30035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:30.017207 30035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "2ed7a39f98614765bac58797f024e29c"
format_stamp: "Formatted at 2026-08-08 06:02:27 on dist-test-slave-zkpd"
server_key: "772c78c1ce152eee3d056e1a04860557"
server_key_iv: "a04f943b6a09d8ad85dc3683632a021a"
server_key_version: "encryptionkey@0"
I20260808 06:02:30.017807 30035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:30.042618 30035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:30.046336 30035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:30.046572 30035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:30.047245 30035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:30.048224 30035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:30.048302 30035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.048396 30035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:30.048450 30035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.057832 30035 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35603
I20260808 06:02:30.057875 30170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35603 every 8 connection(s)
I20260808 06:02:30.058885 30035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:30.060724 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30035
I20260808 06:02:30.061167 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:46723
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:38117
--webserver_port=42185
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:02:30.063503 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.061215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:42651)
0808 06:02:30.061838 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.062478 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":150,"threads_started":1}
W20260808 06:02:30.064066 30171 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38117 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38117: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
I20260808 06:02:30.064774 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.064551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:39639)
0808 06:02:30.064669 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.064718 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 06:02:30.065573 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.065321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:50577)
0808 06:02:30.065459 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.065504 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 06:02:30.065820 30171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:02:30.176087 30174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:30.176383 30174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:30.176463 30174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:30.180397 30174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:30.180486 30174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:30.180575 30174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:30.185616 30174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:46723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=42185
--tserver_master_addrs=127.13.229.126:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:30.186617 30174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:30.187613 30174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:30.197247 30179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:30.197320 30182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:30.197263 30180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:30.197603 30174 server_base.cc:1034] running on GCE node
I20260808 06:02:30.198000 30174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:30.198655 30174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:30.199834 30174 hybrid_clock.cc:648] HybridClock initialized: now 1786168950199797 us; error 48 us; skew 500 ppm
Aug 08 06:02:30 dist-test-slave-zkpd krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168950, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:30.202628 30174 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:30.203490 30174 webserver.cc:460] Webserver started at http://127.13.229.67:42185/ using document root <none> and password file <none>
I20260808 06:02:30.204123 30174 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:30.204218 30174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:30.208112 30174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 06:02:30.210177 30189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.211442 30174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:02:30.211587 30174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "bbc0df7ed8804d72966491202ec85158"
format_stamp: "Formatted at 2026-08-08 06:02:28 on dist-test-slave-zkpd"
server_key: "9d3574acd9a31e04a974ea2d897de3ab"
server_key_iv: "6169fe63276ac958969ff63f8447905d"
server_key_version: "encryptionkey@0"
I20260808 06:02:30.212173 30174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:30.234750 30174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:30.238211 30174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:30.238425 30174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:30.239042 30174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:30.239941 30174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:30.240018 30174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.240089 30174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:30.240139 30174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:30.249213 30174 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:46723
I20260808 06:02:30.249257 30306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:46723 every 8 connection(s)
I20260808 06:02:30.250259 30174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:02:30.254037 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.252391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:44333)
0808 06:02:30.252869 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.253294 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
W20260808 06:02:30.254483 30307 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38117 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38117: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
I20260808 06:02:30.255113 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.254877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:37847)
0808 06:02:30.255004 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.255058 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 06:02:30.255712 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.255501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:40005)
0808 06:02:30.255609 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.255654 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 06:02:30.255882 30307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 06:02:30.259689 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30174
I20260808 06:02:30.919869 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:30.919452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:39721)
0808 06:02:30.919698 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:30.919751 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260808 06:02:31.066730 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.066341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:39357)
0808 06:02:31.066581 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:31.066632 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260808 06:02:31.256831 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.256459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:53845)
0808 06:02:31.256685 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:31.256737 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260808 06:02:31.720510 30314 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:31.720757 30314 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:31.721998 30314 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
Aug 08 06:02:31 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.66@KRBTEST.COM
Aug 08 06:02:31 dist-test-slave-zkpd krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168949, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.67@KRBTEST.COM
I20260808 06:02:31.736840 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.722759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:56113 (local address 127.13.229.67:46723)
0808 06:02:31.723069 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:31.723096 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:31.723144 (+ 48us) server_negotiation.cc:407] Connection header received
0808 06:02:31.723874 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:31.723880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:31.724310 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:31.724602 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:31.725863 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.726808 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728047 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.728511 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728685 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:31.731647 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:31.731655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:31.731671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:31.731707 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:31.733571 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734333 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:31.734339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:31.734345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:31.734494 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734922 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:31.734925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:31.734927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:31.735423 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:31.735548 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:31.736364 (+ 816us) server_negotiation.cc:299] Negotiation successful
0808 06:02:31.736646 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260808 06:02:31.736835 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.722363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:34717 (local address 127.13.229.66:35603)
0808 06:02:31.722931 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:31.722948 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:31.722996 (+ 48us) server_negotiation.cc:407] Connection header received
0808 06:02:31.723748 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:31.723754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:31.724213 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:31.724534 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:31.725832 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.726810 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728052 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.728507 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728692 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:31.731357 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:31.731363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:31.731376 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:31.731403 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:31.733571 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734333 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:31.734339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:31.734345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:31.734494 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734922 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:31.734926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:31.734927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:31.735413 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:31.735548 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:31.736364 (+ 816us) server_negotiation.cc:299] Negotiation successful
0808 06:02:31.736646 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":137,"threads_started":1}
I20260808 06:02:31.736811 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.722215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35603 (local address 127.13.229.65:34717)
0808 06:02:31.722458 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:31.722475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:31.722576 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:31.724602 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:31.724607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:31.725180 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:31.725662 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:31.725677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.726964 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.726967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:31.727882 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:31.727905 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.728636 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:31.728903 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:31.729532 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:31.729547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:31.731107 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:31.733699 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:31.733705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:31.733719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:31.734216 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:31.734599 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:31.734604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:31.734809 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:31.735611 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:31.735617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:31.735986 (+ 369us) client_negotiation.cc:769] Sending connection context
0808 06:02:31.736217 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 06:02:31.736578 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":208,"spinlock_wait_cycles":40576,"thread_start_us":70,"threads_started":1}
I20260808 06:02:31.736812 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.722578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:46723 (local address 127.13.229.65:56113)
0808 06:02:31.722895 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:31.722910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:31.723033 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:31.724623 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:31.724626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:31.725179 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:31.725663 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:31.725678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.726951 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.726954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:31.727879 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:31.727933 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:31.728639 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:31.728643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:31.728907 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:31.729591 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:31.729603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:31.731463 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:31.733699 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:31.733705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:31.733719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:31.734216 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:31.734599 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:31.734603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:31.734604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:31.734809 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:31.735611 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:31.735617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:31.735998 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 06:02:31.736217 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 06:02:31.736631 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":295,"spinlock_wait_cycles":76544,"thread_start_us":77,"threads_started":1}
W20260808 06:02:31.740015 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
W20260808 06:02:31.740630 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:31.740701 29912 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157; no voters: 2ed7a39f98614765bac58797f024e29c, bbc0df7ed8804d72966491202ec85158
I20260808 06:02:31.740923 30314 raft_consensus.cc:2726] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:02:31.921015 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:31.920671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:43975)
0808 06:02:31.920817 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:31.920885 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 06:02:32.068027 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:32.067513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:46065)
0808 06:02:32.067794 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:32.067854 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260808 06:02:32.258004 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:32.257624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:44169)
0808 06:02:32.257865 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:32.257918 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260808 06:02:32.922500 30321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:32.921868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:39121)
0808 06:02:32.922250 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:32.922322 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":83,"threads_started":1}
I20260808 06:02:33.069281 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:33.068890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:37267)
0808 06:02:33.069135 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:33.069190 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260808 06:02:33.259109 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:33.258722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:37237)
0808 06:02:33.258951 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:33.259004 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260808 06:02:33.267931 30324 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:33.268069 30324 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:33.268515 30324 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
W20260808 06:02:33.269459 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
W20260808 06:02:33.269629 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:33.269699 29912 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157; no voters: 2ed7a39f98614765bac58797f024e29c, bbc0df7ed8804d72966491202ec85158
I20260808 06:02:33.269906 30324 raft_consensus.cc:2726] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:02:33.923749 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:33.923313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:56365)
0808 06:02:33.923576 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:33.923628 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260808 06:02:34.070608 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:34.070093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:50351)
0808 06:02:34.070457 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:34.070516 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260808 06:02:34.260221 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:34.259788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:36021)
0808 06:02:34.260073 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:34.260131 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260808 06:02:34.925199 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:34.924738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:55071)
0808 06:02:34.925045 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:34.925102 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260808 06:02:35.071786 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:35.071366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:54131)
0808 06:02:35.071639 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:35.071693 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260808 06:02:35.261366 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:35.260953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:47037)
0808 06:02:35.261216 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:35.261272 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260808 06:02:35.926419 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:35.926006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:47735)
0808 06:02:35.926263 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:35.926319 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260808 06:02:36.072980 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:36.072566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:47571)
0808 06:02:36.072827 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:36.072886 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260808 06:02:36.204608 30333 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:36.204789 30333 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:36.205209 30333 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
W20260808 06:02:36.206087 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
W20260808 06:02:36.206228 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:36.206266 29912 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157; no voters: 2ed7a39f98614765bac58797f024e29c, bbc0df7ed8804d72966491202ec85158
I20260808 06:02:36.206404 30333 raft_consensus.cc:2726] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:02:36.262665 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:36.262203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:40367)
0808 06:02:36.262509 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:36.262571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260808 06:02:36.927541 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:36.927094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:37303)
0808 06:02:36.927384 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:36.927443 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260808 06:02:37.074342 30336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:37.073841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:46183)
0808 06:02:37.074181 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:37.074247 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":76,"threads_started":1}
I20260808 06:02:37.264093 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:37.263609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:44855)
0808 06:02:37.263953 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:37.264004 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":71,"threads_started":1}
I20260808 06:02:37.928735 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:37.928325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:43043)
0808 06:02:37.928582 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:37.928636 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260808 06:02:38.075487 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:38.075060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:57913)
0808 06:02:38.075335 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:38.075393 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260808 06:02:38.265165 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:38.264767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:41689)
0808 06:02:38.265021 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:38.265076 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260808 06:02:38.929796 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:38.929409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:60833)
0808 06:02:38.929648 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:38.929700 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260808 06:02:39.076771 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:39.076334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:57153)
0808 06:02:39.076614 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:39.076674 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260808 06:02:39.266390 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:39.266001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:52919)
0808 06:02:39.266245 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:39.266299 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260808 06:02:39.520818 30344 raft_consensus.cc:482] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:39.520999 30344 raft_consensus.cc:504] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ed7a39f98614765bac58797f024e29c" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35603 } } peers { permanent_uuid: "bbc0df7ed8804d72966491202ec85158" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 46723 } } peers { permanent_uuid: "428cf9b04e904ee3aa5993eadd42c157" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 34737 } }
I20260808 06:02:39.521474 30344 leader_election.cc:288] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603), bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723)
W20260808 06:02:39.522259 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ed7a39f98614765bac58797f024e29c (127.13.229.66:35603): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
W20260808 06:02:39.522425 29912 leader_election.cc:341] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbc0df7ed8804d72966491202ec85158 (127.13.229.67:46723): Not found: Tablet not found: 2d8f75f7e68f4708b2459c4823dfa3a0
I20260808 06:02:39.522482 29912 leader_election.cc:302] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 428cf9b04e904ee3aa5993eadd42c157; no voters: 2ed7a39f98614765bac58797f024e29c, bbc0df7ed8804d72966491202ec85158
I20260808 06:02:39.522651 30344 raft_consensus.cc:2726] T 2d8f75f7e68f4708b2459c4823dfa3a0 P 428cf9b04e904ee3aa5993eadd42c157 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:02:39.931067 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:39.930642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:43103)
0808 06:02:39.930905 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:39.930968 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260808 06:02:40.078157 30346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:40.077565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:35553)
0808 06:02:40.077935 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:40.078027 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":91,"threads_started":1}
I20260808 06:02:40.267553 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:40.267141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:46887)
0808 06:02:40.267406 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:40.267463 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260808 06:02:40.932212 30348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:40.931774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:43753)
0808 06:02:40.932066 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:40.932117 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260808 06:02:41.079413 30350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:41.078887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:36681)
0808 06:02:41.079258 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:41.079316 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":85,"threads_started":1}
I20260808 06:02:41.268746 30351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:41.268345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:49957)
0808 06:02:41.268598 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:41.268653 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260808 06:02:41.933316 30352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:41.932919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:39013)
0808 06:02:41.933165 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:41.933218 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260808 06:02:42.080792 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:42.080392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:39575)
0808 06:02:42.080646 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:42.080697 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260808 06:02:42.269869 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:42.269488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:41121)
0808 06:02:42.269725 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:42.269778 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260808 06:02:42.934492 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:42.934048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:54481)
0808 06:02:42.934334 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:42.934392 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260808 06:02:43.082006 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:43.081552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:33911)
0808 06:02:43.081823 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:43.081909 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260808 06:02:43.271037 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:43.270650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:60143)
0808 06:02:43.270890 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:43.270944 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260808 06:02:43.935765 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:43.935336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:45611)
0808 06:02:43.935614 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:43.935669 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260808 06:02:44.083292 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:44.082759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:39803)
0808 06:02:44.083128 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:44.083193 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260808 06:02:44.272284 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:44.271798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:38197)
0808 06:02:44.272118 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:44.272172 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":138,"threads_started":1}
I20260808 06:02:44.937088 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:44.936669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.65:50679)
0808 06:02:44.936945 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:44.936993 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260808 06:02:45.084667 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:45.084259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.66:34635)
0808 06:02:45.084523 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:45.084575 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260808 06:02:45.268607 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 29860
I20260808 06:02:45.273608 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:45.273216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38117 (local address 127.13.229.67:48837)
0808 06:02:45.273455 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:45.273516 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260808 06:02:45.277513 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30035
I20260808 06:02:45.284518 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30174
2026-08-08T06:02:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19189 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:02:45 dist-test-slave-zkpd krb5kdc[30367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:02:45 dist-test-slave-zkpd krb5kdc[30367](info): set up 2 sockets
Aug 08 06:02:45 dist-test-slave-zkpd krb5kdc[30367](info): commencing operation
krb5kdc: starting...
W20260808 06:02:46.349761 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:02:46 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:02:46.476591 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:46.586889 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:46.587133 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:46.587172 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:46.591176 30381 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 06:02:46.591241 30381 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 06:02:46.591256 30381 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 06:02:46.591270 30381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:02:46.591288 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:46.591363 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:46.591388 30381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:02:46.591418 30381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:02:46.596750 30381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:46.597801 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:46.598872 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:46.607592 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:46.607589 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:46.607656 30381 server_base.cc:1034] running on GCE node
W20260808 06:02:46.607591 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:46.608371 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:46.609416 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:46.610612 30381 hybrid_clock.cc:648] HybridClock initialized: now 1786168966610602 us; error 29 us; skew 500 ppm
Aug 08 06:02:46 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168966, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:46.613459 30381 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:02:46.614369 30381 webserver.cc:460] Webserver started at http://127.13.229.126:40893/ using document root <none> and password file <none>
I20260808 06:02:46.614940 30381 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:46.615029 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:46.615263 30381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:46.617359 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "3fdd13d6539b47abe730b8bb2d52d140"
server_key_iv: "76389ed17ab81bb6d5c5f9b635e21a41"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.617944 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "3fdd13d6539b47abe730b8bb2d52d140"
server_key_iv: "76389ed17ab81bb6d5c5f9b635e21a41"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.621631 30381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:02:46.623968 30397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:46.625108 30381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:46.625249 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "3fdd13d6539b47abe730b8bb2d52d140"
server_key_iv: "76389ed17ab81bb6d5c5f9b635e21a41"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.625360 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:46.642248 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:46.645586 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:46.645808 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:46.653136 30381 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38937
I20260808 06:02:46.653445 30449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38937 every 8 connection(s)
I20260808 06:02:46.654141 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:02:46.657058 30450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:46.662504 30450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap starting.
I20260808 06:02:46.662971 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30381
I20260808 06:02:46.663146 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:02:46.663429 14229 external_mini_cluster.cc:1428] Setting key 15f739fc79b16d81cd1a92910778fb6a
I20260808 06:02:46.665309 30450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:46.666082 30450 log.cc:824] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:46.668321 30450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: No bootstrap required, opened a new log
Aug 08 06:02:46 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:46.672995 30450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:02:46.673270 30450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:46.673344 30450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Initialized, Role: FOLLOWER
I20260808 06:02:46.673902 30450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:02:46.674088 30450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:02:46.674171 30450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:02:46.674296 30450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:46.675180 30450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:02:46.675530 30450 leader_election.cc:302] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350ee7b0a4764ec38aed2e3eb2bf7a39; no voters:
I20260808 06:02:46.675895 30450 leader_election.cc:288] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:02:46.675988 30455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:46.676210 30455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 LEADER]: Becoming Leader. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Running, Role: LEADER
I20260808 06:02:46.676717 30455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:02:46.677259 30450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:02:46.678474 30457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:02:46.678560 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:46.664782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.13.229.126:38937)
0808 06:02:46.665200 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:46.665215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:46.665251 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:02:46.665934 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:46.665952 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:46.666321 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:46.666677 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:46.667529 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.668649 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.669789 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.670233 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.670347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:46.673203 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:46.673210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:46.673235 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:46.673269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:46.675117 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:46.675612 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:46.675618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:46.675624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:46.675788 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:46.676131 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:46.676142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:46.676144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:46.676662 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:46.676890 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:46.677193 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 06:02:46.677625 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260808 06:02:46.678601 30457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
I20260808 06:02:46.678951 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350ee7b0a4764ec38aed2e3eb2bf7a39. Latest consensus state: current_term: 1 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:02:46.679041 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
W20260808 06:02:46.682392 30471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:02:46.682493 30471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:02:46.682571 30464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:02:46.683266 30464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:02:46.688269 30464 catalog_manager.cc:1347] Generated new cluster ID: 9aa459e28f5749f6a9e332d27a7b6d56
I20260808 06:02:46.688331 30464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:02:46.700338 30464 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:02:46.701530 30464 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:02:46.706533 30464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Generated new TSK 0
I20260808 06:02:46.707265 30464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:02:46.742599 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:46.858035 30476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:46.858297 30476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:46.858340 30476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:46.862308 30476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:46.862361 30476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:46.862418 30476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:02:46.867403 30476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:46.868379 30476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:46.869417 30476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:46.879175 30481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:46.879243 30482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:46.879369 30476 server_base.cc:1034] running on GCE node
W20260808 06:02:46.879216 30484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:46.879878 30476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:46.880564 30476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:46.881750 30476 hybrid_clock.cc:648] HybridClock initialized: now 1786168966881717 us; error 45 us; skew 500 ppm
Aug 08 06:02:46 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168966, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:46.884465 30476 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:02:46.885285 30476 webserver.cc:460] Webserver started at http://127.13.229.65:35397/ using document root <none> and password file <none>
I20260808 06:02:46.885838 30476 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:46.885931 30476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:46.886157 30476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:46.888217 30476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "b4c2880cb53e45b8868b8ea17cc56343"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "75482f5925a4ea4cd9d15209f327bb1d"
server_key_iv: "41383a43b0c52d838b53e9221cd209a7"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.888798 30476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c2880cb53e45b8868b8ea17cc56343"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "75482f5925a4ea4cd9d15209f327bb1d"
server_key_iv: "41383a43b0c52d838b53e9221cd209a7"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.892386 30476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:02:46.894691 30491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:46.895937 30476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:46.896073 30476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "b4c2880cb53e45b8868b8ea17cc56343"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "75482f5925a4ea4cd9d15209f327bb1d"
server_key_iv: "41383a43b0c52d838b53e9221cd209a7"
server_key_version: "encryptionkey@0"
I20260808 06:02:46.896183 30476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:46.916740 30476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:46.920475 30476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:46.920748 30476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:46.921455 30476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:46.922425 30476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:46.922474 30476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:46.922514 30476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:46.922529 30476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:46.931723 30476 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:42863
I20260808 06:02:46.931744 30611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:42863 every 8 connection(s)
I20260808 06:02:46.932665 30476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:02:46.939821 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30476
I20260808 06:02:46.940021 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:02:46.940316 14229 external_mini_cluster.cc:1428] Setting key 5f6205730f8ec066f3fb7823d90d9137
Aug 08 06:02:46 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168966, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:46.946940 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:46.934528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:55965 (local address 127.13.229.126:38937)
0808 06:02:46.934668 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:46.934671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:46.935698 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 06:02:46.936795 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:46.936798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:46.936860 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:46.936987 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:46.938172 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.938711 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.939937 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.940255 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.940372 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:46.942872 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:46.942876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:46.942878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:46.942903 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:46.944744 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:46.945234 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:46.945240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:46.945245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:46.945336 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:46.945677 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:46.945683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:46.945688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:46.945940 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:46.946047 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:46.946526 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 06:02:46.946682 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:02:46.947556 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:46.934868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:55965)
0808 06:02:46.935549 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:46.935591 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:46.936556 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:46.937167 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:46.937174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:46.937550 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:46.938026 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:46.938040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.938837 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.938840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:46.939775 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:46.939786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:46.940416 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:46.940420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:46.940599 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:46.941165 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:46.941181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:46.942715 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:46.944866 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:46.944873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:46.944885 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:46.945146 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:46.945445 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:46.945447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:46.945448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:46.945581 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:46.946046 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:46.946050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:46.946274 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 06:02:46.946434 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 06:02:46.946678 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":118,"threads_started":1}
I20260808 06:02:46.948879 30612 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:02:46.949139 30612 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:46.949642 30612 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:02:46.951190 30414 ts_manager.cc:194] Registered new tserver with Master: b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65:42863)
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:02:46.952448 30414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:55965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:02:46.970896 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:47.086555 30617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:47.086942 30617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:47.086995 30617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:47.090979 30617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:47.091032 30617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:47.091089 30617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:02:47.096223 30617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:47.097294 30617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:47.098275 30617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:47.107998 30623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:47.108028 30622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:47.108114 30617 server_base.cc:1034] running on GCE node
W20260808 06:02:47.107998 30625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:47.108626 30617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:47.109274 30617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:47.110469 30617 hybrid_clock.cc:648] HybridClock initialized: now 1786168967110459 us; error 28 us; skew 500 ppm
Aug 08 06:02:47 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168967, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:47.113274 30617 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:02:47.114140 30617 webserver.cc:460] Webserver started at http://127.13.229.66:40769/ using document root <none> and password file <none>
I20260808 06:02:47.114707 30617 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:47.114801 30617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:47.115046 30617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:47.117127 30617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "90161dd4ad2844c68a0cda92820a1fee"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "3718bf30595e98e0a7ed651640c3af56"
server_key_iv: "a64af6056e1fb941875bb09d6649e02a"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.117712 30617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "90161dd4ad2844c68a0cda92820a1fee"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "3718bf30595e98e0a7ed651640c3af56"
server_key_iv: "a64af6056e1fb941875bb09d6649e02a"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.121371 30617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:02:47.123699 30632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.124949 30617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:02:47.125092 30617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "90161dd4ad2844c68a0cda92820a1fee"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "3718bf30595e98e0a7ed651640c3af56"
server_key_iv: "a64af6056e1fb941875bb09d6649e02a"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.125203 30617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:47.140635 30617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:47.144106 30617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:47.144320 30617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:47.144954 30617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:47.145831 30617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:47.145906 30617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.145985 30617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:47.146039 30617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.155409 30617 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:42779
I20260808 06:02:47.155437 30751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:42779 every 8 connection(s)
I20260808 06:02:47.156337 30617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:02:47.158015 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30617
I20260808 06:02:47.158175 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:02:47.158470 14229 external_mini_cluster.cc:1428] Setting key 1d32951a7374b2ca8dc74f3c6ae9857c
Aug 08 06:02:47 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168967, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:02:47.171690 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.158210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:39407 (local address 127.13.229.126:38937)
0808 06:02:47.158371 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.158376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.159534 (+ 1158us) server_negotiation.cc:407] Connection header received
0808 06:02:47.160789 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.160791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.160834 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.160902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:47.162564 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.163422 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.164640 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.165084 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.165216 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.168118 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:47.168121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:47.168123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:47.168147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:47.169698 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.170176 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.170178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.170180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.170252 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.170606 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.170609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.170610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.170789 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:47.170870 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.171358 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.171516 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:02:47.172509 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.158624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:39407)
0808 06:02:47.159388 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.159432 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.160553 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.161091 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.161103 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.161647 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:47.162389 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.162413 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.163551 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.163554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.164513 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.164524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.165246 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.165249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.165465 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.166079 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:47.166096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:47.167941 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:47.169808 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:47.169813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:47.169824 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:47.170080 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:47.170376 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:47.170379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:47.170380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:47.170514 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:47.170884 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:47.170889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:47.171125 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.171287 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.171536 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":157,"threads_started":1}
I20260808 06:02:47.173784 30752 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:02:47.174043 30752 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:47.174554 30752 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:02:47.175722 30414 ts_manager.cc:194] Registered new tserver with Master: 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
I20260808 06:02:47.176620 30414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:39407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:02:47.190470 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:02:47.305195 30757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:02:47.305491 30757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:02:47.305570 30757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:02:47.309729 30757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:02:47.309810 30757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:02:47.309973 30757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:02:47.315239 30757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:02:47.316386 30757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:02:47.317420 30757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:02:47.327121 30764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:47.327186 30757 server_base.cc:1034] running on GCE node
W20260808 06:02:47.327147 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:02:47.327139 30766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:02:47.327788 30757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:02:47.328616 30757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:02:47.329818 30757 hybrid_clock.cc:648] HybridClock initialized: now 1786168967329781 us; error 44 us; skew 500 ppm
Aug 08 06:02:47 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168967, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:02:47.332597 30757 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:02:47.333416 30757 webserver.cc:460] Webserver started at http://127.13.229.67:41447/ using document root <none> and password file <none>
I20260808 06:02:47.333956 30757 fs_manager.cc:359] Metadata directory not provided
I20260808 06:02:47.334028 30757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:02:47.334195 30757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:02:47.336783 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "d5700911098345998da25f265826fe59"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "74e8c1b3b2cae9971a596cbf26afbe9d"
server_key_iv: "37ce6488a89a989ba26012fb02839b4b"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.337394 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "d5700911098345998da25f265826fe59"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "74e8c1b3b2cae9971a596cbf26afbe9d"
server_key_iv: "37ce6488a89a989ba26012fb02839b4b"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.341115 30757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:02:47.343461 30773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.344813 30757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:02:47.344952 30757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "d5700911098345998da25f265826fe59"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "74e8c1b3b2cae9971a596cbf26afbe9d"
server_key_iv: "37ce6488a89a989ba26012fb02839b4b"
server_key_version: "encryptionkey@0"
I20260808 06:02:47.345062 30757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:02:47.362497 30757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:02:47.365406 30757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:02:47.365630 30757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:02:47.366246 30757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:02:47.367097 30757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:02:47.367146 30757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.367218 30757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:02:47.367259 30757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:02:47.376705 30757 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41397
I20260808 06:02:47.376806 30890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41397 every 8 connection(s)
I20260808 06:02:47.377732 30757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
Aug 08 06:02:47 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168967, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:02:47.387444 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30757
I20260808 06:02:47.387579 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:02:47.387880 14229 external_mini_cluster.cc:1428] Setting key 5ec2eb9998e0c3bd307346950c8594b7
I20260808 06:02:47.391624 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.379652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:54307 (local address 127.13.229.126:38937)
0808 06:02:47.379797 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.379801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.380937 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 06:02:47.381836 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.381839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.381918 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.382025 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:47.383185 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.383722 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.384912 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.385139 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.385258 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.387930 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:47.387935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:47.387938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:47.387969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:47.389600 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.390115 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.390120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.390122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.390209 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.390549 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.390554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.390556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.390739 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:47.390846 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.391329 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.391458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:02:47.392540 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.380155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:54307)
0808 06:02:47.380794 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.380835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.381629 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.382202 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.382209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.382568 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:02:47.383040 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.383053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.383854 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.383862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.384777 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.384788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.385294 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.385298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.385469 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.386007 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:02:47.386023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:02:47.387733 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:02:47.389725 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:47.389731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:47.389744 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:47.390010 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:47.390301 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:02:47.390305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:02:47.390307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:02:47.390455 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:02:47.390845 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:02:47.390851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:02:47.391095 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.391274 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.391585 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":106,"threads_started":1}
I20260808 06:02:47.393580 30891 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:02:47.393842 30891 heartbeater.cc:464] Registering TS with master...
I20260808 06:02:47.394341 30891 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:02:47.395349 30414 ts_manager.cc:194] Registered new tserver with Master: d5700911098345998da25f265826fe59 (127.13.229.67:41397)
I20260808 06:02:47.395982 30414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:54307
I20260808 06:02:47.401713 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:02:47.410797 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.403662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.13.229.126:38937)
0808 06:02:47.403785 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.403788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.404025 (+ 237us) server_negotiation.cc:407] Connection header received
0808 06:02:47.404126 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.404128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.404165 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.404233 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:02:47.405049 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.405551 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.406318 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.406526 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.406598 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.408048 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:02:47.408050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:02:47.408052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:02:47.408073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:02:47.409493 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.409847 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.409849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.409850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.409921 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:02:47.410145 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:02:47.410147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:02:47.410148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:02:47.410340 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:02:47.410448 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.410598 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.410684 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:02:47.413643 30414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:02:47.416723 30414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:02:47.431592 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.425662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.13.229.66:42779)
0808 06:02:47.426388 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.426402 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.426421 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:02:47.426484 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.426487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.426629 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.426802 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:47.427304 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.428528 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.430269 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.430877 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.431006 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.431162 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.431329 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.431450 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":165,"threads_started":1}
I20260808 06:02:47.431583 30903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.425083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42779 (local address 127.0.0.1:42834)
0808 06:02:47.425923 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.425936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.426023 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.426948 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.426950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.426980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:47.427167 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.427173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.428684 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.428687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.430135 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.430147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.431006 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.431009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.431118 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.431157 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.431298 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.431422 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":139,"spinlock_wait_cycles":14592,"thread_start_us":117,"threads_started":1}
I20260808 06:02:47.431823 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.424870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42863 (local address 127.0.0.1:52204)
0808 06:02:47.425370 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.425418 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.425548 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.426796 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.426803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.426826 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:47.426978 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.426984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.428684 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.428687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.430238 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.430252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.431394 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.431398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.431485 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.431517 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.431576 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.431663 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":155,"thread_start_us":69,"threads_started":1}
I20260808 06:02:47.432663 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.425070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.13.229.65:42863)
0808 06:02:47.426269 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.426283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.426308 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:02:47.426389 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.426395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.426601 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.426772 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:47.427105 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.428520 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.430368 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.431242 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.432184 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.432344 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.432399 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.432495 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":123,"threads_started":1}
I20260808 06:02:47.432852 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.425544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41397 (local address 127.0.0.1:57454)
0808 06:02:47.426121 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.426136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.426227 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.426944 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.426948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.426982 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:47.427166 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.427173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.429330 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.429333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.430563 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.430576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.432437 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.432440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.432522 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.432546 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.432626 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.432705 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":79,"spinlock_wait_cycles":11392,"thread_start_us":101,"threads_started":1}
I20260808 06:02:47.432915 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.425913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.13.229.67:41397)
0808 06:02:47.426392 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.426403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.426421 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:02:47.426483 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.426488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.426628 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.426772 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:47.427989 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.429180 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.430703 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.432153 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.432262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.432373 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.432677 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.432773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":169,"threads_started":1}
I20260808 06:02:47.435042 30683 tablet_service.cc:1467] Processing CreateTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 (DEFAULT_TABLE table=test-table [id=57b7e0f23183478880ed4330a795d3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:47.435192 30541 tablet_service.cc:1467] Processing CreateTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 (DEFAULT_TABLE table=test-table [id=57b7e0f23183478880ed4330a795d3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:47.436264 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08d4c148e6b4276a11488c9e91f94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:47.436275 30541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08d4c148e6b4276a11488c9e91f94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:47.438357 30822 tablet_service.cc:1467] Processing CreateTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 (DEFAULT_TABLE table=test-table [id=57b7e0f23183478880ed4330a795d3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:02:47.439378 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08d4c148e6b4276a11488c9e91f94a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:02:47.442677 30908 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Bootstrap starting.
I20260808 06:02:47.444103 30909 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap starting.
I20260808 06:02:47.445785 30910 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Bootstrap starting.
I20260808 06:02:47.446244 30908 tablet_bootstrap.cc:654] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:47.446468 30909 tablet_bootstrap.cc:654] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:47.447302 30908 log.cc:824] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:47.447607 30910 tablet_bootstrap.cc:654] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Neither blocks nor log segments found. Creating new log.
I20260808 06:02:47.447763 30909 log.cc:824] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:47.448935 30910 log.cc:824] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Log is configured to *not* fsync() on all Append() calls
I20260808 06:02:47.449808 30908 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: No bootstrap required, opened a new log
I20260808 06:02:47.449842 30909 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: No bootstrap required, opened a new log
I20260808 06:02:47.450018 30908 ts_tablet_manager.cc:1397] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 06:02:47.450029 30909 ts_tablet_manager.cc:1397] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:02:47.450842 30910 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: No bootstrap required, opened a new log
I20260808 06:02:47.451026 30910 ts_tablet_manager.cc:1397] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 06:02:47.453110 30908 raft_consensus.cc:348] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.453403 30908 raft_consensus.cc:374] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:47.453459 30908 raft_consensus.cc:729] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90161dd4ad2844c68a0cda92820a1fee, State: Initialized, Role: FOLLOWER
I20260808 06:02:47.453944 30908 consensus_queue.cc:260] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.453828 30909 raft_consensus.cc:348] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.454105 30909 raft_consensus.cc:374] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:47.454165 30909 raft_consensus.cc:729] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Initialized, Role: FOLLOWER
I20260808 06:02:47.454566 30909 consensus_queue.cc:260] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.454632 30752 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
I20260808 06:02:47.455013 30908 ts_tablet_manager.cc:1428] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 06:02:47.454833 30910 raft_consensus.cc:348] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.455081 30910 raft_consensus.cc:374] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:02:47.455128 30910 raft_consensus.cc:729] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5700911098345998da25f265826fe59, State: Initialized, Role: FOLLOWER
I20260808 06:02:47.455523 30909 ts_tablet_manager.cc:1428] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:02:47.455596 30612 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
I20260808 06:02:47.455552 30910 consensus_queue.cc:260] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.456557 30910 ts_tablet_manager.cc:1428] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 06:02:47.456904 30891 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
I20260808 06:02:47.541229 30915 raft_consensus.cc:482] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:02:47.541473 30915 raft_consensus.cc:504] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.542585 30915 leader_election.cc:288] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779), d5700911098345998da25f265826fe59 (127.13.229.67:41397)
I20260808 06:02:47.548162 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.543022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40105 (local address 127.13.229.66:42779)
0808 06:02:47.543170 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.543175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.543558 (+ 383us) server_negotiation.cc:407] Connection header received
0808 06:02:47.543640 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.543644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.543701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.543789 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:47.544302 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.545096 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.546901 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.547518 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.547646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.547699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.547948 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.548030 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 06:02:47.548375 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.543130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:54051 (local address 127.13.229.67:41397)
0808 06:02:47.543281 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.543285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.543489 (+ 204us) server_negotiation.cc:407] Connection header received
0808 06:02:47.543645 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.543648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.543697 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.543786 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:02:47.544302 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.545092 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.546901 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.547519 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.547646 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.547699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.548166 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.548247 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:02:47.548389 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.542886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42779 (local address 127.13.229.65:40105)
0808 06:02:47.543454 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.543468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.543549 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.543818 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.543821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.543877 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:47.544169 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.544177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.545212 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.545215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.546760 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.546773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.547662 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.547668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.547761 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.547789 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.547873 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.548020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":365,"spinlock_wait_cycles":35328,"thread_start_us":94,"threads_started":1}
I20260808 06:02:47.548820 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.542876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41397 (local address 127.13.229.65:54051)
0808 06:02:47.543379 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.543396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:02:47.543541 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:02:47.543818 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:02:47.543821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:02:47.543879 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:02:47.544173 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.544178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.545212 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.545215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.546760 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:02:47.546773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.547910 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.547916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:02:47.548019 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.548042 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:02:47.548615 (+ 573us) client_negotiation.cc:240] Negotiation successful
0808 06:02:47.548687 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":166,"spinlock_wait_cycles":8064,"thread_start_us":129,"threads_started":1}
I20260808 06:02:47.549266 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08d4c148e6b4276a11488c9e91f94a5" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90161dd4ad2844c68a0cda92820a1fee" is_pre_election: true
I20260808 06:02:47.549584 30842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08d4c148e6b4276a11488c9e91f94a5" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5700911098345998da25f265826fe59" is_pre_election: true
I20260808 06:02:47.549667 30703 raft_consensus.cc:2454] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 0.
I20260808 06:02:47.549911 30842 raft_consensus.cc:2454] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 0.
I20260808 06:02:47.550269 30493 leader_election.cc:302] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90161dd4ad2844c68a0cda92820a1fee, b4c2880cb53e45b8868b8ea17cc56343; no voters:
I20260808 06:02:47.550601 30915 raft_consensus.cc:2781] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:02:47.550694 30915 raft_consensus.cc:482] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:02:47.550738 30915 raft_consensus.cc:3037] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:47.551828 30915 raft_consensus.cc:504] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.552211 30915 leader_election.cc:288] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 election: Requested vote from peers 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779), d5700911098345998da25f265826fe59 (127.13.229.67:41397)
I20260808 06:02:47.552580 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08d4c148e6b4276a11488c9e91f94a5" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90161dd4ad2844c68a0cda92820a1fee"
I20260808 06:02:47.552629 30842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08d4c148e6b4276a11488c9e91f94a5" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5700911098345998da25f265826fe59"
I20260808 06:02:47.552698 30703 raft_consensus.cc:3037] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:47.552788 30842 raft_consensus.cc:3037] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:02:47.553797 30703 raft_consensus.cc:2454] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 1.
I20260808 06:02:47.553911 30842 raft_consensus.cc:2454] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 1.
I20260808 06:02:47.554114 30493 leader_election.cc:302] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90161dd4ad2844c68a0cda92820a1fee, b4c2880cb53e45b8868b8ea17cc56343; no voters:
I20260808 06:02:47.554344 30915 raft_consensus.cc:2781] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:02:47.554577 30915 raft_consensus.cc:686] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 LEADER]: Becoming Leader. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Running, Role: LEADER
I20260808 06:02:47.554924 30915 consensus_queue.cc:237] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:02:47.558110 30413 catalog_manager.cc:5495] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "b4c2880cb53e45b8868b8ea17cc56343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } health_report { overall_health: UNKNOWN } } }
I20260808 06:02:47.586959 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.583405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.13.229.65:42863)
0808 06:02:47.583555 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:02:47.583562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:02:47.583578 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:02:47.583668 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:02:47.583671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:02:47.583727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:02:47.583851 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:02:47.584134 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.584634 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.585493 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:02:47.585713 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:02:47.585830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:02:47.586139 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:02:47.586560 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:02:47.586685 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 06:02:47.586752 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 06:02:47.586808 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:02:47.592973 30703 raft_consensus.cc:1261] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Refusing update from remote peer b4c2880cb53e45b8868b8ea17cc56343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:47.592973 30842 raft_consensus.cc:1261] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Refusing update from remote peer b4c2880cb53e45b8868b8ea17cc56343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:02:47.593777 30915 consensus_queue.cc:1035] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:47.594022 30919 consensus_queue.cc:1035] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:02:47.602692 30923 mvcc.cc:205] Tried to move back new op lower bound from 7316148091252715520 to 7316148091107598336. Current Snapshot: MvccSnapshot[applied={T|T < 7316148091252715520}]
I20260808 06:02:47.604228 30924 mvcc.cc:205] Tried to move back new op lower bound from 7316148091252715520 to 7316148091107598336. Current Snapshot: MvccSnapshot[applied={T|T < 7316148091252715520 or (T in {7316148091252715520})}]
I20260808 06:02:47.609333 30413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:47.609509 30413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:02:47.611949 30413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Sending DeleteTablet for 3 replicas of tablet c08d4c148e6b4276a11488c9e91f94a5
I20260808 06:02:47.612694 30683 tablet_service.cc:1514] Processing DeleteTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:42834
I20260808 06:02:47.612694 30822 tablet_service.cc:1514] Processing DeleteTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:57454
I20260808 06:02:47.612699 30541 tablet_service.cc:1514] Processing DeleteTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:52204
I20260808 06:02:47.613225 30933 tablet_replica.cc:330] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: stopping tablet replica
I20260808 06:02:47.613404 30931 tablet_replica.cc:330] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: stopping tablet replica
I20260808 06:02:47.613473 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30476
I20260808 06:02:47.617969 30932 tablet_replica.cc:330] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: stopping tablet replica
I20260808 06:02:47.618292 30932 raft_consensus.cc:2229] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:47.618623 30932 raft_consensus.cc:2258] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:47.619016 30931 raft_consensus.cc:2229] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:02:47.619318 30931 raft_consensus.cc:2258] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:02:47.620913 30931 ts_tablet_manager.cc:1905] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:47.623131 30931 ts_tablet_manager.cc:1918] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:47.623253 30931 log.cc:1196] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/c08d4c148e6b4276a11488c9e91f94a5
I20260808 06:02:47.623544 30931 ts_tablet_manager.cc:1939] T c08d4c148e6b4276a11488c9e91f94a5 P 90161dd4ad2844c68a0cda92820a1fee: Deleting consensus metadata
I20260808 06:02:47.625064 30399 catalog_manager.cc:4841] TS 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): tablet c08d4c148e6b4276a11488c9e91f94a5 (table test-table [id=57b7e0f23183478880ed4330a795d3a3]) successfully deleted
W20260808 06:02:47.626448 30398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:42863 (error 108)
I20260808 06:02:47.626786 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30617
I20260808 06:02:47.629550 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:02:47.629178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42863 (local address 127.0.0.1:52224)
0808 06:02:47.629344 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 06:02:47.629440 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 06:02:47.629737 30398 catalog_manager.cc:4584] TS b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65:42863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c08d4c148e6b4276a11488c9e91f94a5: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42863: connect: Connection refused (error 111)
I20260808 06:02:47.630750 30932 ts_tablet_manager.cc:1905] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:02:47.633112 30932 ts_tablet_manager.cc:1918] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:02:47.633196 30932 log.cc:1196] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/c08d4c148e6b4276a11488c9e91f94a5
I20260808 06:02:47.633522 30932 ts_tablet_manager.cc:1939] T c08d4c148e6b4276a11488c9e91f94a5 P d5700911098345998da25f265826fe59: Deleting consensus metadata
I20260808 06:02:47.634652 30400 catalog_manager.cc:4841] TS d5700911098345998da25f265826fe59 (127.13.229.67:41397): tablet c08d4c148e6b4276a11488c9e91f94a5 (table test-table [id=57b7e0f23183478880ed4330a795d3a3]) successfully deleted
I20260808 06:02:47.635622 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30757
I20260808 06:02:47.645174 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30381
I20260808 06:03:07.653039 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--webserver_interface=127.13.229.126
--webserver_port=40893
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:07.767908 30935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:07.768182 30935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:07.768224 30935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:07.772269 30935 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 06:03:07.772315 30935 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 06:03:07.772331 30935 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 06:03:07.772346 30935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:07.772363 30935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:07.772382 30935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:07.772396 30935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:07.772411 30935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:07.777567 30935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=40893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:07.778673 30935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:07.779673 30935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:07.794107 30944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:07.794185 30935 server_base.cc:1034] running on GCE node
W20260808 06:03:07.794114 30942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:07.794121 30941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:07.795024 30935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:07.796314 30935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:07.797513 30935 hybrid_clock.cc:648] HybridClock initialized: now 1786168987797497 us; error 27 us; skew 500 ppm
Aug 08 06:03:07 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:07.800361 30935 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:03:07.801244 30935 webserver.cc:460] Webserver started at http://127.13.229.126:40893/ using document root <none> and password file <none>
I20260808 06:03:07.801837 30935 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:07.801964 30935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:07.806035 30935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:07.808228 30951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:07.809458 30935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:03:07.809605 30935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "3fdd13d6539b47abe730b8bb2d52d140"
server_key_iv: "76389ed17ab81bb6d5c5f9b635e21a41"
server_key_version: "encryptionkey@0"
I20260808 06:03:07.810210 30935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:07.821532 30935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:07.824443 30935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:07.824664 30935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:07.832264 30935 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38937
I20260808 06:03:07.832566 31004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38937 every 8 connection(s)
I20260808 06:03:07.833412 30935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:07.837879 31005 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:03:07.839623 31005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap starting.
I20260808 06:03:07.841163 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 30935
I20260808 06:03:07.841689 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:42863
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=35397
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:03:07.858687 31005 log.cc:824] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:07.867692 31005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:03:07.868129 31005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap complete.
I20260808 06:03:07.872339 31005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:07.872772 31005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Initialized, Role: FOLLOWER
I20260808 06:03:07.873318 31005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:07.873477 31005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:07.873574 31005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:07.873695 31005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 1 FOLLOWER]: Advancing to term 2
I20260808 06:03:07.875617 31005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:07.876041 31005 leader_election.cc:302] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350ee7b0a4764ec38aed2e3eb2bf7a39; no voters:
I20260808 06:03:07.876415 31005 leader_election.cc:288] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 06:03:07.876502 31011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 06:03:07.876765 31011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 LEADER]: Becoming Leader. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Running, Role: LEADER
I20260808 06:03:07.877135 31011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:07.877717 31005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:07.878129 31011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350ee7b0a4764ec38aed2e3eb2bf7a39. Latest consensus state: current_term: 2 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:03:07.878242 31011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:07.878620 31015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:07.878707 31012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:03:07.878893 31012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:07.883451 31015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:07.884151 31015 catalog_manager.cc:1259] Loaded cluster ID: 9aa459e28f5749f6a9e332d27a7b6d56
I20260808 06:03:07.884219 31015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:07.886679 31015 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:07.887430 31015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Loaded TSK: 0
I20260808 06:03:07.906814 31015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Generated new TSK 1
I20260808 06:03:07.907109 31015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:03:07.961040 31009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:07.961436 31009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:07.961531 31009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:07.965566 31009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:07.965619 31009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:07.965711 31009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:07.971027 31009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:42863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=35397
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:07.972102 31009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:07.973085 31009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:07.983857 31036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:07.983875 31035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:07.983875 31038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:07.984874 31009 server_base.cc:1034] running on GCE node
I20260808 06:03:07.985291 31009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:07.986008 31009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:07.987205 31009 hybrid_clock.cc:648] HybridClock initialized: now 1786168987987173 us; error 49 us; skew 500 ppm
Aug 08 06:03:07 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:07.990329 31009 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:03:07.991298 31009 webserver.cc:460] Webserver started at http://127.13.229.65:35397/ using document root <none> and password file <none>
I20260808 06:03:07.991883 31009 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:07.992057 31009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:07.997148 31009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 06:03:07.999540 31045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.000977 31009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:03:08.001122 31009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "b4c2880cb53e45b8868b8ea17cc56343"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "75482f5925a4ea4cd9d15209f327bb1d"
server_key_iv: "41383a43b0c52d838b53e9221cd209a7"
server_key_version: "encryptionkey@0"
I20260808 06:03:08.001717 31009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:08.023548 31009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:08.026821 31009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:08.027045 31009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:08.027695 31009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:08.028978 31052 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:03:08.033272 31009 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:03:08.033342 31009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260808 06:03:08.033427 31009 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:03:08.035162 31009 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:03:08.035226 31009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 06:03:08.035494 31052 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap starting.
I20260808 06:03:08.046761 31009 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:42863
I20260808 06:03:08.046775 31161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:42863 every 8 connection(s)
I20260808 06:03:08.048055 31009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:08.049861 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31009
I20260808 06:03:08.050523 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:42779
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=40769
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:03:08.059010 31052 log.cc:824] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:08.067492 31052 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:03:08.067968 31052 tablet_bootstrap.cc:492] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap complete.
I20260808 06:03:08.068698 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.050212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:40323)
0808 06:03:08.051264 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.051307 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.052204 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.053626 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.053635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.054137 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:08.054907 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.054932 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.056366 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.056372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.057793 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.057805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.058819 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.058826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.059109 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.060024 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:08.060052 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:08.062057 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:08.065043 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.065049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.065061 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.065415 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.065873 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.065875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.065877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.066047 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.066756 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:08.066763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:08.067049 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.067254 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.067606 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":176,"threads_started":1}
I20260808 06:03:08.069094 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.050215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40323 (local address 127.13.229.126:38937)
0808 06:03:08.050966 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.050986 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.051430 (+ 444us) server_negotiation.cc:407] Connection header received
0808 06:03:08.052672 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.052690 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.053103 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.053484 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.055113 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.056162 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.057945 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.058649 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.058819 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.062294 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.062304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.062327 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.062366 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.064898 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.065555 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.065564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.065573 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.065750 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.066159 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.066162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.066164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.066621 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.066757 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.067391 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.067797 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":120,"threads_started":1}
I20260808 06:03:08.069581 31052 ts_tablet_manager.cc:1397] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.016s
I20260808 06:03:08.071692 31162 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:03:08.071965 31162 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:08.072528 31162 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:03:08.072244 31052 raft_consensus.cc:348] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:03:08.072755 31052 raft_consensus.cc:729] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Initialized, Role: FOLLOWER
I20260808 06:03:08.073241 31052 consensus_queue.cc:260] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } }
I20260808 06:03:08.074275 31052 ts_tablet_manager.cc:1428] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 06:03:08.076001 30969 ts_manager.cc:194] Registered new tserver with Master: b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65:42863)
I20260808 06:03:08.080317 30969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:40323
I20260808 06:03:08.083014 31162 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168987, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.65@KRBTEST.COM
I20260808 06:03:08.094437 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.079355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42863 (local address 127.0.0.1:53882)
0808 06:03:08.079725 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.079771 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.079927 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.080570 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.080576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.080930 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:08.081097 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.081107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.082387 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.082391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.083308 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.083320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.084081 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.084085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.084194 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.089012 (+ 4818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:08.089027 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:08.090533 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:08.092547 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.092558 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.092564 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.092802 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.093131 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.093134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.093136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.093226 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.093714 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:08.093720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:08.093869 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.094023 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.094237 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":121,"threads_started":1}
I20260808 06:03:08.094439 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.079403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.13.229.65:42863)
0808 06:03:08.080176 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.080184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.080201 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:03:08.080250 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.080254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.080396 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.080557 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.081235 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.082255 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.083441 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.083944 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.084065 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.090690 (+ 6625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.090705 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.090715 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.090737 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.092419 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.092928 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.092936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.092938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.093025 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.093322 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.093327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.093328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.093584 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.093751 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.094109 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.094276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":95,"threads_started":1}
I20260808 06:03:08.095554 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet c08d4c148e6b4276a11488c9e91f94a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:02:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53882
I20260808 06:03:08.096666 31172 tablet_replica.cc:330] stopping tablet replica
I20260808 06:03:08.096976 31172 raft_consensus.cc:2229] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:08.097235 31172 raft_consensus.cc:2258] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:08.098228 31172 ts_tablet_manager.cc:1905] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:08.102094 31172 ts_tablet_manager.cc:1918] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:08.102206 31172 log.cc:1196] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal/wals/c08d4c148e6b4276a11488c9e91f94a5
I20260808 06:03:08.102480 31172 ts_tablet_manager.cc:1939] T c08d4c148e6b4276a11488c9e91f94a5 P b4c2880cb53e45b8868b8ea17cc56343: Deleting consensus metadata
I20260808 06:03:08.104295 30952 catalog_manager.cc:4841] TS b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65:42863): tablet c08d4c148e6b4276a11488c9e91f94a5 (table test-table [id=57b7e0f23183478880ed4330a795d3a3]) successfully deleted
W20260808 06:03:08.175279 31167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:08.175535 31167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:08.175577 31167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:08.180037 31167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:08.180094 31167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:08.180193 31167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:08.185307 31167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:42779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=40769
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:08.186316 31167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:08.187330 31167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:08.197216 31178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:08.197211 31181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:08.197259 31179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:08.197685 31167 server_base.cc:1034] running on GCE node
I20260808 06:03:08.198101 31167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:08.198729 31167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:08.199944 31167 hybrid_clock.cc:648] HybridClock initialized: now 1786168988199885 us; error 38 us; skew 500 ppm
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168988, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:08.202953 31167 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:03:08.203851 31167 webserver.cc:460] Webserver started at http://127.13.229.66:40769/ using document root <none> and password file <none>
I20260808 06:03:08.204463 31167 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:08.204572 31167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:08.208683 31167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:08.210760 31188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.212041 31167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:08.212173 31167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "90161dd4ad2844c68a0cda92820a1fee"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "3718bf30595e98e0a7ed651640c3af56"
server_key_iv: "a64af6056e1fb941875bb09d6649e02a"
server_key_version: "encryptionkey@0"
I20260808 06:03:08.212706 31167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:08.223734 31167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:08.226979 31167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:08.227177 31167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:08.227734 31167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:08.228595 31167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:08.228670 31167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.228735 31167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:08.228783 31167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.237628 31167 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:42779
I20260808 06:03:08.237706 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:42779 every 8 connection(s)
I20260808 06:03:08.238624 31167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:08.240726 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31167
I20260808 06:03:08.241187 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:41397
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=41447
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168988, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:08.255689 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.240674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:36183 (local address 127.13.229.126:38937)
0808 06:03:08.240857 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.240862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.242154 (+ 1292us) server_negotiation.cc:407] Connection header received
0808 06:03:08.243560 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.243566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.243638 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.243752 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.244959 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.245934 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.247201 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.247714 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.247864 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.251199 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.251202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.251205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.251239 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.252987 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.253819 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.253823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.253824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.253909 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.254408 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.254412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.254413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.254603 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.254691 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.255449 (+ 758us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.255564 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 06:03:08.256933 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.241128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:36183)
0808 06:03:08.241960 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.242020 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.243272 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.243961 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.243974 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.244310 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:08.244780 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.244793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.246081 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.246084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.247044 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.247054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.248070 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.248073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.248224 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.248938 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:08.248954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:08.250978 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:08.253130 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.253141 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.253159 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.253614 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.254027 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.254034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.254038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.254275 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.254734 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:08.254742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:08.255128 (+ 386us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.255401 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.255819 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":154,"threads_started":1}
I20260808 06:03:08.258225 31305 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:03:08.258488 31305 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:08.258988 31305 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:03:08.260089 30969 ts_manager.cc:194] Registered new tserver with Master: 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
I20260808 06:03:08.260654 30969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:36183
W20260808 06:03:08.361443 31308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:08.361739 31308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:08.361828 31308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:08.365737 31308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:08.365839 31308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:08.365927 31308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:08.371011 31308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:41397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=41447
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:08.372109 31308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:08.373169 31308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:08.383045 31317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:08.383086 31313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:08.383086 31314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:08.383279 31308 server_base.cc:1034] running on GCE node
I20260808 06:03:08.383742 31308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:08.384423 31308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:08.385586 31308 hybrid_clock.cc:648] HybridClock initialized: now 1786168988385574 us; error 24 us; skew 500 ppm
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168988, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:08.388434 31308 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:03:08.389344 31308 webserver.cc:460] Webserver started at http://127.13.229.67:41447/ using document root <none> and password file <none>
I20260808 06:03:08.389845 31308 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:08.389915 31308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:08.393957 31308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:08.395954 31324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.397145 31308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:08.397281 31308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "d5700911098345998da25f265826fe59"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "74e8c1b3b2cae9971a596cbf26afbe9d"
server_key_iv: "37ce6488a89a989ba26012fb02839b4b"
server_key_version: "encryptionkey@0"
I20260808 06:03:08.397820 31308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:08.442122 31308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:08.445555 31308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:08.445783 31308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:08.446416 31308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:08.447263 31308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:08.447336 31308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.447409 31308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:08.447459 31308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:08.456456 31308 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41397
I20260808 06:03:08.456493 31443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41397 every 8 connection(s)
I20260808 06:03:08.457460 31308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:08.460314 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31308
Aug 08 06:03:08 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786168988, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:08.469786 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.461069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.13.229.126:38937)
0808 06:03:08.461325 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.461329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.461431 (+ 102us) server_negotiation.cc:407] Connection header received
0808 06:03:08.461538 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.461541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.461596 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.461697 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.462434 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.463232 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.464259 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.464586 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.464689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.466090 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.466093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.466096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.466121 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.467672 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.468210 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.468213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.468214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.468317 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.468645 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.468649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.468651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.468921 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.469225 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.469322 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.469490 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
I20260808 06:03:08.472887 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.459339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:51289 (local address 127.13.229.126:38937)
0808 06:03:08.459504 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.459507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.460797 (+ 1290us) server_negotiation.cc:407] Connection header received
0808 06:03:08.461947 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.461950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.461993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.462069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.463540 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.464090 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.465460 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.465666 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.465777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.468532 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.468535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.468537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.468559 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.470341 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.471094 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.471097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.471099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.471251 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.471662 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.471666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.471683 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.471939 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.472047 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.472681 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.472783 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 06:03:08.473832 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.459723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:51289)
0808 06:03:08.460577 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.460634 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.461718 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.462245 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.462251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.462734 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:08.463373 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.463398 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.464229 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.464235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.465310 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.465320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.465822 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.465826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.466002 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.466553 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:08.466580 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:08.468333 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:08.470492 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.470498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.470515 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.470916 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.471374 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:08.471379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:08.471383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:08.471569 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:08.472110 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:08.472120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:08.472417 (+ 297us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.472594 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.472870 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":157,"threads_started":1}
I20260808 06:03:08.474996 31444 heartbeater.cc:347] Connected to a master server at 127.13.229.126:38937
I20260808 06:03:08.475261 31444 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:08.475786 31444 heartbeater.cc:511] Master 127.13.229.126:38937 requested a full tablet report, sending...
I20260808 06:03:08.476919 30969 ts_manager.cc:194] Registered new tserver with Master: d5700911098345998da25f265826fe59 (127.13.229.67:41397)
I20260808 06:03:08.477500 30969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:51289
I20260808 06:03:08.483847 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:08.487515 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.484474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.13.229.126:38937)
0808 06:03:08.484585 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.484590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.484787 (+ 197us) server_negotiation.cc:407] Connection header received
0808 06:03:08.484925 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.484927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.485021 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.485101 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:08.485304 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.485775 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.486572 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.486768 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.486846 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.487116 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:08.487238 (+ 122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 06:03:08.487428 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45788: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 06:03:08.487599 31165 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45788: token expired
W20260808 06:03:08.488015 30895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.229.126:38937)
I20260808 06:03:08.495412 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.488398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.13.229.126:38937)
0808 06:03:08.488531 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.488536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.488554 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:03:08.488641 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.488644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.488680 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.488772 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:08.489460 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.489927 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.490744 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.490958 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.491066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.492482 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:08.492486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:08.492489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:08.492513 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:08.493779 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.494201 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.494205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.494207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.494291 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:08.494596 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:08.494599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:08.494600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:08.494771 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:08.494907 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.495143 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.495252 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:03:08.497756 30969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:08.500975 30969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:08.509025 31094 tablet_service.cc:1467] Processing CreateTablet for tablet 041a9b5424cf4f08833e2905ce71f663 (DEFAULT_TABLE table=test-table [id=b90bf01d2fa845829a14c6f573911b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:08.509655 31094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a9b5424cf4f08833e2905ce71f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:08.512768 31052 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap starting.
I20260808 06:03:08.513691 31052 tablet_bootstrap.cc:654] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:08.513744 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.507950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41397 (local address 127.0.0.1:51014)
0808 06:03:08.508087 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.508102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.508181 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.509067 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.509070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.509089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:08.509198 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.509204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.510886 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.510890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.512094 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.512110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.513227 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.513230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.513328 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.513378 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.513466 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.513544 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260808 06:03:08.513852 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.508132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.13.229.67:41397)
0808 06:03:08.508559 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.508573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.508599 (+ 26us) server_negotiation.cc:407] Connection header received
0808 06:03:08.508680 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.508685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.508901 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.509077 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:08.509334 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.510771 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.512209 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.513098 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.513216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.513364 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.513540 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.513633 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":181,"threads_started":1}
I20260808 06:03:08.514415 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.508213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42779 (local address 127.0.0.1:47332)
0808 06:03:08.508718 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.508733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.508832 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.509624 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.509627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.509646 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:08.509828 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.509836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.511158 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.511162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.512027 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.512041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.513907 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.513911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.514011 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.514042 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.514200 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.514264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":115,"threads_started":1}
I20260808 06:03:08.514623 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.508658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.13.229.66:42779)
0808 06:03:08.509121 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.509133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.509154 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:03:08.509231 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.509239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.509462 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.509631 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:08.509962 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.511042 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.512237 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.513098 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.513216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.513364 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.514236 (+ 872us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.514332 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":123,"threads_started":1}
I20260808 06:03:08.515247 31052 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: No bootstrap required, opened a new log
I20260808 06:03:08.515347 31052 ts_tablet_manager.cc:1397] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 06:03:08.515852 31052 raft_consensus.cc:348] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.516016 31052 raft_consensus.cc:374] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:08.516062 31052 raft_consensus.cc:729] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Initialized, Role: FOLLOWER
I20260808 06:03:08.516218 31052 consensus_queue.cc:260] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.516572 31052 ts_tablet_manager.cc:1428] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 06:03:08.517004 31375 tablet_service.cc:1467] Processing CreateTablet for tablet 041a9b5424cf4f08833e2905ce71f663 (DEFAULT_TABLE table=test-table [id=b90bf01d2fa845829a14c6f573911b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:08.517577 31236 tablet_service.cc:1467] Processing CreateTablet for tablet 041a9b5424cf4f08833e2905ce71f663 (DEFAULT_TABLE table=test-table [id=b90bf01d2fa845829a14c6f573911b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:08.518167 31375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a9b5424cf4f08833e2905ce71f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:08.518616 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a9b5424cf4f08833e2905ce71f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:08.527205 31459 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Bootstrap starting.
I20260808 06:03:08.526703 31458 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Bootstrap starting.
I20260808 06:03:08.529686 31459 tablet_bootstrap.cc:654] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:08.530048 31458 tablet_bootstrap.cc:654] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:08.530495 31459 log.cc:824] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:08.530776 31458 log.cc:824] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:08.532707 31458 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: No bootstrap required, opened a new log
I20260808 06:03:08.532707 31459 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: No bootstrap required, opened a new log
I20260808 06:03:08.532912 31458 ts_tablet_manager.cc:1397] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:08.532912 31459 ts_tablet_manager.cc:1397] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 06:03:08.537012 31459 raft_consensus.cc:348] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.537012 31458 raft_consensus.cc:348] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.537321 31459 raft_consensus.cc:374] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:08.537321 31458 raft_consensus.cc:374] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:08.537386 31458 raft_consensus.cc:729] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5700911098345998da25f265826fe59, State: Initialized, Role: FOLLOWER
I20260808 06:03:08.537386 31459 raft_consensus.cc:729] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90161dd4ad2844c68a0cda92820a1fee, State: Initialized, Role: FOLLOWER
I20260808 06:03:08.537889 31459 consensus_queue.cc:260] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.537889 31458 consensus_queue.cc:260] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.538604 31444 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
I20260808 06:03:08.538980 31458 ts_tablet_manager.cc:1428] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:03:08.538980 31459 ts_tablet_manager.cc:1428] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:03:08.539305 31305 heartbeater.cc:503] Master 127.13.229.126:38937 was elected leader, sending a full tablet report...
W20260808 06:03:08.549477 31164 tablet.cc:2369] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:03:08.708412 31445 tablet.cc:2369] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:08.728220 31169 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:08.728510 31169 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.729712 31169 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
I20260808 06:03:08.733827 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.730232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:55563 (local address 127.13.229.66:42779)
0808 06:03:08.730371 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.730376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.730588 (+ 212us) server_negotiation.cc:407] Connection header received
0808 06:03:08.730751 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.730765 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.730812 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.730906 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:08.731180 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.731667 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.732803 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.733198 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.733338 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.733392 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.733586 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.733661 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 06:03:08.733829 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.730127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:41517 (local address 127.13.229.67:41397)
0808 06:03:08.730280 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.730286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.730518 (+ 232us) server_negotiation.cc:407] Connection header received
0808 06:03:08.730693 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.730697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.730742 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.730830 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:08.731130 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.731635 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.732806 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.733218 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.733353 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.733404 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.733596 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.733673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:03:08.733866 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.730005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41397 (local address 127.13.229.65:41517)
0808 06:03:08.730408 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.730422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.730538 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.730841 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.730845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.730861 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:08.731012 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.731019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.731753 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.731757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.732664 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.732674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.733350 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.733353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.733469 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.733488 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.733580 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.733690 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":53,"thread_start_us":104,"threads_started":1}
I20260808 06:03:08.733908 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.730005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42779 (local address 127.13.229.65:55563)
0808 06:03:08.730498 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.730516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:08.730636 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:08.730920 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:08.730923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:08.730939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:08.731067 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.731072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.731793 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.731798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.732664 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:08.732673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.733331 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.733335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:08.733424 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.733470 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 06:03:08.733571 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 06:03:08.733667 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":166,"thread_start_us":66,"threads_started":1}
I20260808 06:03:08.734488 31395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a9b5424cf4f08833e2905ce71f663" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5700911098345998da25f265826fe59" is_pre_election: true
I20260808 06:03:08.734583 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a9b5424cf4f08833e2905ce71f663" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90161dd4ad2844c68a0cda92820a1fee" is_pre_election: true
I20260808 06:03:08.734839 31258 raft_consensus.cc:2454] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 0.
I20260808 06:03:08.734839 31395 raft_consensus.cc:2454] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 0.
I20260808 06:03:08.735357 31048 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343, d5700911098345998da25f265826fe59; no voters:
I20260808 06:03:08.735606 31169 raft_consensus.cc:2781] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:08.735685 31169 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:08.735718 31169 raft_consensus.cc:3037] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:08.736840 31169 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.737192 31169 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 election: Requested vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
I20260808 06:03:08.737543 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a9b5424cf4f08833e2905ce71f663" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90161dd4ad2844c68a0cda92820a1fee"
I20260808 06:03:08.737622 31395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a9b5424cf4f08833e2905ce71f663" candidate_uuid: "b4c2880cb53e45b8868b8ea17cc56343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5700911098345998da25f265826fe59"
I20260808 06:03:08.737694 31258 raft_consensus.cc:3037] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:08.737766 31395 raft_consensus.cc:3037] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:08.738775 31395 raft_consensus.cc:2454] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 1.
I20260808 06:03:08.738782 31258 raft_consensus.cc:2454] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c2880cb53e45b8868b8ea17cc56343 in term 1.
I20260808 06:03:08.739075 31048 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343, d5700911098345998da25f265826fe59; no voters:
I20260808 06:03:08.739282 31169 raft_consensus.cc:2781] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:08.739524 31169 raft_consensus.cc:686] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 LEADER]: Becoming Leader. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Running, Role: LEADER
W20260808 06:03:08.739765 31306 tablet.cc:2369] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:08.739789 31169 consensus_queue.cc:237] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:08.742735 30968 catalog_manager.cc:5495] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "b4c2880cb53e45b8868b8ea17cc56343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } health_report { overall_health: HEALTHY } } }
I20260808 06:03:08.760855 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.757234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.13.229.65:42863)
0808 06:03:08.757544 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:08.757558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:08.757569 (+ 11us) server_negotiation.cc:407] Connection header received
0808 06:03:08.757648 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:08.757651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:08.757700 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:08.757824 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:08.758083 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.758582 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.759421 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:08.759645 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:08.759759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:08.760054 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:08.760456 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:08.760588 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:08.760651 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 06:03:08.760703 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":117,"threads_started":1}
I20260808 06:03:08.764648 31395 raft_consensus.cc:1261] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Refusing update from remote peer b4c2880cb53e45b8868b8ea17cc56343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:08.764671 31258 raft_consensus.cc:1261] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Refusing update from remote peer b4c2880cb53e45b8868b8ea17cc56343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:08.765362 31466 consensus_queue.cc:1035] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:08.765556 31169 consensus_queue.cc:1035] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:08.774884 31470 mvcc.cc:205] Tried to move back new op lower bound from 7316148177973506048 to 7316148177881047040. Current Snapshot: MvccSnapshot[applied={T|T < 7316148177973506048}]
I20260808 06:03:08.782078 30969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:08.782223 30969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:08.784492 30969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Sending DeleteTablet for 3 replicas of tablet 041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:08.785241 31375 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a9b5424cf4f08833e2905ce71f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:08 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:51014
I20260808 06:03:08.785241 31236 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a9b5424cf4f08833e2905ce71f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:08 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:47332
I20260808 06:03:08.785929 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31009
I20260808 06:03:08.786027 31094 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a9b5424cf4f08833e2905ce71f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:08 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:53882
I20260808 06:03:08.788242 31480 tablet_replica.cc:330] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: stopping tablet replica
I20260808 06:03:08.788573 31480 raft_consensus.cc:2229] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:08.788913 31480 raft_consensus.cc:2258] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:08.789202 31481 tablet_replica.cc:330] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: stopping tablet replica
I20260808 06:03:08.789723 31481 raft_consensus.cc:2229] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:08.790043 31481 raft_consensus.cc:2258] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:08.790508 31480 ts_tablet_manager.cc:1905] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:08.791419 31481 ts_tablet_manager.cc:1905] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:08.793538 31481 ts_tablet_manager.cc:1918] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:08.793658 31481 log.cc:1196] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:08.794114 31481 ts_tablet_manager.cc:1939] T 041a9b5424cf4f08833e2905ce71f663 P d5700911098345998da25f265826fe59: Deleting consensus metadata
I20260808 06:03:08.794176 31480 ts_tablet_manager.cc:1918] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:08.794286 31480 log.cc:1196] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:08.794621 31480 ts_tablet_manager.cc:1939] T 041a9b5424cf4f08833e2905ce71f663 P 90161dd4ad2844c68a0cda92820a1fee: Deleting consensus metadata
I20260808 06:03:08.795341 30955 catalog_manager.cc:4841] TS d5700911098345998da25f265826fe59 (127.13.229.67:41397): tablet 041a9b5424cf4f08833e2905ce71f663 (table test-table [id=b90bf01d2fa845829a14c6f573911b6f]) successfully deleted
I20260808 06:03:08.795845 30954 catalog_manager.cc:4841] TS 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): tablet 041a9b5424cf4f08833e2905ce71f663 (table test-table [id=b90bf01d2fa845829a14c6f573911b6f]) successfully deleted
W20260808 06:03:08.796490 30952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:42863 (error 108)
I20260808 06:03:08.797380 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31167
I20260808 06:03:08.797605 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:08.797269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42863 (local address 127.0.0.1:53906)
0808 06:03:08.797412 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:08.797511 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 06:03:08.803712 30952 catalog_manager.cc:4584] TS b4c2880cb53e45b8868b8ea17cc56343 (127.13.229.65:42863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 041a9b5424cf4f08833e2905ce71f663: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42863: connect: Connection refused (error 111)
I20260808 06:03:08.805984 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31308
I20260808 06:03:08.813131 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 30935
I20260808 06:03:28.820969 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--webserver_interface=127.13.229.126
--webserver_port=40893
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:28.938941 31483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:28.939203 31483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:28.939244 31483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:28.943357 31483 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 06:03:28.943409 31483 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 06:03:28.943425 31483 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 06:03:28.943440 31483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:28.943459 31483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:28.943476 31483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:28.943492 31483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:28.943507 31483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:28.948757 31483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=40893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:28.949715 31483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:28.950660 31483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:28.959909 31489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:28.959949 31488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:28.959951 31491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:28.960000 31483 server_base.cc:1034] running on GCE node
I20260808 06:03:28.960834 31483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:28.961892 31483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:28.963104 31483 hybrid_clock.cc:648] HybridClock initialized: now 1786169008963092 us; error 31 us; skew 500 ppm
Aug 08 06:03:28 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:28.970760 31483 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:03:28.971776 31483 webserver.cc:460] Webserver started at http://127.13.229.126:40893/ using document root <none> and password file <none>
I20260808 06:03:28.972440 31483 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:28.972541 31483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:28.976603 31483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:28.978782 31498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:28.980120 31483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:28.980258 31483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "3fdd13d6539b47abe730b8bb2d52d140"
server_key_iv: "76389ed17ab81bb6d5c5f9b635e21a41"
server_key_version: "encryptionkey@0"
I20260808 06:03:28.980829 31483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:28.994251 31483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:28.997196 31483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:28.997395 31483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:29.005183 31483 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38937
I20260808 06:03:29.005507 31553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38937 every 8 connection(s)
I20260808 06:03:29.006278 31483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:29.009364 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31483
I20260808 06:03:29.009835 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:42863
--local_ip_for_outbound_sockets=127.13.229.65
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=35397
--webserver_interface=127.13.229.65
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:03:29.011032 31554 sys_catalog.cc:263] Verifying existing consensus state
I20260808 06:03:29.012897 31554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap starting.
I20260808 06:03:29.032747 31554 log.cc:824] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:29.045302 31554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:03:29.045730 31554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Bootstrap complete.
I20260808 06:03:29.050346 31554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:29.050807 31554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Initialized, Role: FOLLOWER
I20260808 06:03:29.051335 31554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:29.051497 31554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:29.051592 31554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:29.051730 31554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 2 FOLLOWER]: Advancing to term 3
I20260808 06:03:29.053834 31554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:29.054199 31554 leader_election.cc:302] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 350ee7b0a4764ec38aed2e3eb2bf7a39; no voters:
I20260808 06:03:29.054548 31554 leader_election.cc:288] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 06:03:29.054689 31558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 06:03:29.054960 31558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [term 3 LEADER]: Becoming Leader. State: Replica: 350ee7b0a4764ec38aed2e3eb2bf7a39, State: Running, Role: LEADER
I20260808 06:03:29.055339 31558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } }
I20260808 06:03:29.055929 31554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:29.056915 31559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:03:29.057065 31559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:29.057461 31560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 350ee7b0a4764ec38aed2e3eb2bf7a39. Latest consensus state: current_term: 3 leader_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "350ee7b0a4764ec38aed2e3eb2bf7a39" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38937 } } }
I20260808 06:03:29.057610 31560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:29.057921 31567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:29.062995 31567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:29.063680 31567 catalog_manager.cc:1259] Loaded cluster ID: 9aa459e28f5749f6a9e332d27a7b6d56
I20260808 06:03:29.063740 31567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:29.066243 31567 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:29.066996 31567 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Loaded TSK: 0
I20260808 06:03:29.067078 31567 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 350ee7b0a4764ec38aed2e3eb2bf7a39: Loaded TSK: 1
F20260808 06:03:29.067847 31567 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f68f11429ad google::LogMessage::Fail() at ??:0
@ 0x7f68f114588a google::LogMessage::SendToLog() at ??:0
@ 0x7f68f11424de google::LogMessage::Flush() at ??:0
@ 0x7f68f1143a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f68f3e0618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f68f3dfef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f68fbaa77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f68fba8199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f68fba82178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f68fbac3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f68fbae282b std::function<>::operator()() at ??:0
@ 0x7f68fba81b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f68fba82f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f68fba7f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f68fbac3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68f33943d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f68f3394c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f68f33963f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f68f3385512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f68f49c96db start_thread at ??:0
@ 0x7f68efc4061f clone at ??:0
W20260808 06:03:29.154857 31556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:29.155100 31556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:29.155146 31556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:29.159567 31556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:29.159618 31556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:29.159680 31556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:29.165418 31556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:42863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=35397
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:29.166664 31556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:29.167732 31556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:29.177953 31600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.177963 31603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.178145 31601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:29.178700 31556 server_base.cc:1034] running on GCE node
I20260808 06:03:29.179119 31556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:29.179786 31556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:29.180961 31556 hybrid_clock.cc:648] HybridClock initialized: now 1786169009180928 us; error 45 us; skew 500 ppm
Aug 08 06:03:29 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:29.184091 31556 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:03:29.184940 31556 webserver.cc:460] Webserver started at http://127.13.229.65:35397/ using document root <none> and password file <none>
I20260808 06:03:29.185501 31556 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:29.185563 31556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:29.189548 31556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:29.191838 31613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.193188 31556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:03:29.193325 31556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "b4c2880cb53e45b8868b8ea17cc56343"
format_stamp: "Formatted at 2026-08-08 06:02:46 on dist-test-slave-zkpd"
server_key: "75482f5925a4ea4cd9d15209f327bb1d"
server_key_iv: "41383a43b0c52d838b53e9221cd209a7"
server_key_version: "encryptionkey@0"
I20260808 06:03:29.193909 31556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:29.214504 31556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:29.218057 31556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:29.218308 31556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:29.218966 31556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:29.220199 31621 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 06:03:29.225764 31556 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 06:03:29.225844 31556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 06:03:29.225929 31556 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 06:03:29.227622 31556 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 06:03:29.227691 31556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 06:03:29.228071 31621 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap starting.
I20260808 06:03:29.238869 31556 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:42863
I20260808 06:03:29.239154 31730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:42863 every 8 connection(s)
I20260808 06:03:29.239948 31556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:29.248950 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31556
I20260808 06:03:29.249445 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:42779
--local_ip_for_outbound_sockets=127.13.229.66
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=40769
--webserver_interface=127.13.229.66
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:03:29.251272 31621 log.cc:824] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:29.260702 31621 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 06:03:29.261168 31621 tablet_bootstrap.cc:492] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Bootstrap complete.
I20260808 06:03:29.262852 31621 ts_tablet_manager.cc:1397] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.013s
I20260808 06:03:29.266444 31621 raft_consensus.cc:348] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:29.267030 31621 raft_consensus.cc:729] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c2880cb53e45b8868b8ea17cc56343, State: Initialized, Role: FOLLOWER
I20260808 06:03:29.267601 31621 consensus_queue.cc:260] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:29.268802 31621 ts_tablet_manager.cc:1428] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260808 06:03:29.325474 31733 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:03:29.242057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:34007)
0808 06:03:29.242554 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.242608 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:29.243431 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:29.324454 (+ 81023us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
W20260808 06:03:29.326129 31731 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38937 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38937: Client connection negotiation failed: client connection to 127.13.229.126:38937: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 06:03:29.327006 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.326701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:47979)
0808 06:03:29.326858 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.326941 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 06:03:29.327771 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.327541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:44697)
0808 06:03:29.327660 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.327714 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 06:03:29.328001 31731 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:03:29.377506 31734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:29.377750 31734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:29.377800 31734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:29.381749 31734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:29.381855 31734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:29.381913 31734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:29.387096 31734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:42779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=40769
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:29.388126 31734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:29.389122 31734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:29.398943 31742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.398973 31744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.399017 31741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:29.399109 31734 server_base.cc:1034] running on GCE node
I20260808 06:03:29.399680 31734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:29.400378 31734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:29.401544 31734 hybrid_clock.cc:648] HybridClock initialized: now 1786169009401530 us; error 29 us; skew 500 ppm
Aug 08 06:03:29 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:29.404552 31734 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:03:29.405305 31734 webserver.cc:460] Webserver started at http://127.13.229.66:40769/ using document root <none> and password file <none>
I20260808 06:03:29.405819 31734 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:29.405879 31734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:29.409662 31734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:29.411653 31752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.412900 31734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:29.413017 31734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "90161dd4ad2844c68a0cda92820a1fee"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "3718bf30595e98e0a7ed651640c3af56"
server_key_iv: "a64af6056e1fb941875bb09d6649e02a"
server_key_version: "encryptionkey@0"
I20260808 06:03:29.413534 31734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:29.439739 31734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:29.443151 31734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:29.443374 31734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:29.444048 31734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:29.444896 31734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:29.444967 31734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.445041 31734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:29.445086 31734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.453991 31734 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:42779
I20260808 06:03:29.454058 31868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:42779 every 8 connection(s)
I20260808 06:03:29.455054 31734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:29.459458 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.457269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:52179)
0808 06:03:29.458231 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.458403 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":560,"threads_started":1}
W20260808 06:03:29.460016 31869 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38937 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38937: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
I20260808 06:03:29.460717 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.460428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:36633)
0808 06:03:29.460574 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.460648 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 06:03:29.460986 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31734
I20260808 06:03:29.461536 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.461142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:42185)
0808 06:03:29.461407 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.461465 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260808 06:03:29.461542 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:41397
--local_ip_for_outbound_sockets=127.13.229.67
--tserver_master_addrs=127.13.229.126:38937
--webserver_port=41447
--webserver_interface=127.13.229.67
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:29.461673 31869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 06:03:29.578306 31873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:29.578588 31873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:29.578660 31873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:29.582753 31873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:29.582846 31873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:29.582933 31873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:29.588229 31873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:41397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=41447
--tserver_master_addrs=127.13.229.126:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:29.589372 31873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:29.590438 31873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:29.600421 31881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.600425 31878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:29.600425 31879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:29.600656 31873 server_base.cc:1034] running on GCE node
I20260808 06:03:29.601179 31873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:29.601766 31873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:29.602928 31873 hybrid_clock.cc:648] HybridClock initialized: now 1786169009602914 us; error 32 us; skew 500 ppm
Aug 08 06:03:29 dist-test-slave-zkpd krb5kdc[30367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:29.605671 31873 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:03:29.606514 31873 webserver.cc:460] Webserver started at http://127.13.229.67:41447/ using document root <none> and password file <none>
I20260808 06:03:29.607106 31873 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:29.607172 31873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:29.611008 31873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:29.613076 31888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.614295 31873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:29.614445 31873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "d5700911098345998da25f265826fe59"
format_stamp: "Formatted at 2026-08-08 06:02:47 on dist-test-slave-zkpd"
server_key: "74e8c1b3b2cae9971a596cbf26afbe9d"
server_key_iv: "37ce6488a89a989ba26012fb02839b4b"
server_key_version: "encryptionkey@0"
I20260808 06:03:29.615008 31873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:29.643429 31873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:29.646431 31873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:29.646643 31873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:29.647286 31873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:29.648216 31873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:29.648288 31873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.648363 31873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:29.648415 31873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:29.657725 31873 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41397
I20260808 06:03:29.657755 32006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41397 every 8 connection(s)
I20260808 06:03:29.658731 31873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:29.660507 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 31873
I20260808 06:03:29.662612 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.660998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:43725)
0808 06:03:29.661747 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.661874 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":151,"threads_started":1}
W20260808 06:03:29.663060 32007 heartbeater.cc:644] Failed to heartbeat to 127.13.229.126:38937 (0 consecutive failures): Network error: Failed to ping master at 127.13.229.126:38937: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
I20260808 06:03:29.663651 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.663432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:43639)
0808 06:03:29.663548 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.663599 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 06:03:29.664275 32009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:29.664076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:46089)
0808 06:03:29.664177 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:29.664223 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:03:29.664469 32007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 06:03:30.329038 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.328604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:60511)
0808 06:03:30.328856 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:30.328923 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260808 06:03:30.462741 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.462246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:58503)
0808 06:03:30.462535 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:30.462613 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260808 06:03:30.559343 32013 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:30.559614 32013 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:30.560916 32013 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
Aug 08 06:03:30 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.66@KRBTEST.COM
Aug 08 06:03:30 dist-test-slave-zkpd krb5kdc[30367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169009, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.67@KRBTEST.COM
I20260808 06:03:30.577916 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.561177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41397 (local address 127.13.229.65:56859)
0808 06:03:30.561405 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:30.561421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:30.561535 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:30.563652 (+ 2117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:30.563657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:30.564317 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:30.565154 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:30.565177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.567088 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.567091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:30.568785 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:30.568798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.569769 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.569772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:30.570103 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:30.570900 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:30.570930 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:30.573233 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:30.575559 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:30.575562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:30.575563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:30.575793 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:30.576224 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:30.576228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:30.576230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:30.576324 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:30.576925 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:30.576928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:30.577184 (+ 256us) client_negotiation.cc:769] Sending connection context
0808 06:03:30.577364 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 06:03:30.577699 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":429,"spinlock_wait_cycles":12416}
I20260808 06:03:30.577929 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.561354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:56859 (local address 127.13.229.67:41397)
0808 06:03:30.561812 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:30.561829 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:30.561874 (+ 45us) server_negotiation.cc:407] Connection header received
0808 06:03:30.562776 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:30.562785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:30.563252 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:30.563592 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:30.565353 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.566927 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.568921 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.569637 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.569810 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:30.573440 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:30.573445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:30.573455 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:30.573483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:30.575430 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:30.575953 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:30.575961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:30.575966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:30.576093 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:30.576431 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:30.576434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:30.576436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:30.576812 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:30.576957 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:30.577497 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 06:03:30.577756 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
I20260808 06:03:30.577973 32015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.561387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:57835 (local address 127.13.229.66:42779)
0808 06:03:30.561812 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:30.561829 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:30.561883 (+ 54us) server_negotiation.cc:407] Connection header received
0808 06:03:30.562776 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:30.562785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:30.563252 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:30.563592 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:30.565353 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.566917 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.568921 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.569642 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.569811 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:30.573110 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:30.573117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:30.573135 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:30.573176 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:30.575121 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:30.575852 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:30.575858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:30.575865 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:30.576017 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:30.576388 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:30.576392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:30.576393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:30.576783 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:30.576931 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:30.577509 (+ 578us) server_negotiation.cc:299] Negotiation successful
0808 06:03:30.577779 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":91,"threads_started":1}
I20260808 06:03:30.577963 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.561216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42779 (local address 127.13.229.65:57835)
0808 06:03:30.561567 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:30.561582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:30.561688 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:30.564002 (+ 2314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:30.564005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:30.564317 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:30.565154 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:30.565177 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.567088 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.567091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:30.568785 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:30.568798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:30.569769 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:30.569772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:30.570105 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:30.570900 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:30.570922 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:30.572847 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:30.575261 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:30.575268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:30.575291 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:30.575710 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:30.576121 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:30.576125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:30.576127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:30.576270 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:30.576911 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:30.576918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:30.577183 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 06:03:30.577363 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 06:03:30.577789 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":514,"spinlock_wait_cycles":32768,"thread_start_us":65,"threads_started":1}
W20260808 06:03:30.580894 31615 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
W20260808 06:03:30.581192 31616 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5700911098345998da25f265826fe59 (127.13.229.67:41397): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:30.581270 31616 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343; no voters: 90161dd4ad2844c68a0cda92820a1fee, d5700911098345998da25f265826fe59
I20260808 06:03:30.581511 32013 raft_consensus.cc:2726] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:03:30.665558 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:30.665109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:46417)
0808 06:03:30.665396 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:30.665463 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260808 06:03:31.330437 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:31.329935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:45045)
0808 06:03:31.330260 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:31.330329 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260808 06:03:31.464160 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:31.463622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:35257)
0808 06:03:31.463943 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:31.464022 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260808 06:03:31.666671 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:31.666262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:38199)
0808 06:03:31.666514 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:31.666567 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260808 06:03:32.331773 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:32.331292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:45857)
0808 06:03:32.331601 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:32.331674 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20260808 06:03:32.436158 32022 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:32.436316 32022 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:32.436738 32022 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
W20260808 06:03:32.437651 31615 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
W20260808 06:03:32.437786 31616 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5700911098345998da25f265826fe59 (127.13.229.67:41397): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:32.437892 31616 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343; no voters: 90161dd4ad2844c68a0cda92820a1fee, d5700911098345998da25f265826fe59
I20260808 06:03:32.438107 32022 raft_consensus.cc:2726] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:03:32.465252 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:32.464820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:43465)
0808 06:03:32.465091 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:32.465143 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260808 06:03:32.667936 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:32.667483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:46995)
0808 06:03:32.667733 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:32.667793 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260808 06:03:33.332983 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:33.332560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:35341)
0808 06:03:33.332826 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:33.332886 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260808 06:03:33.466446 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:33.465997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:46349)
0808 06:03:33.466271 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:33.466346 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260808 06:03:33.669044 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:33.668638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:33023)
0808 06:03:33.668906 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:33.668958 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260808 06:03:34.334525 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:34.334060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:35625)
0808 06:03:34.334372 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:34.334429 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260808 06:03:34.467669 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:34.467236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:52151)
0808 06:03:34.467521 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:34.467575 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
I20260808 06:03:34.670353 32031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:34.669928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:50849)
0808 06:03:34.670202 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:34.670260 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
I20260808 06:03:35.131305 32032 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:35.131453 32032 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:35.131819 32032 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
W20260808 06:03:35.132710 31616 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5700911098345998da25f265826fe59 (127.13.229.67:41397): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
W20260808 06:03:35.132828 31615 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:35.132892 31615 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343; no voters: 90161dd4ad2844c68a0cda92820a1fee, d5700911098345998da25f265826fe59
I20260808 06:03:35.133038 32032 raft_consensus.cc:2726] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:03:35.335707 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:35.335301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:58757)
0808 06:03:35.335552 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:35.335607 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260808 06:03:35.468966 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:35.468551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:47679)
0808 06:03:35.468808 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:35.468869 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260808 06:03:35.671571 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:35.671166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:54735)
0808 06:03:35.671411 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:35.671471 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260808 06:03:36.336897 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:36.336490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:48419)
0808 06:03:36.336745 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:36.336800 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260808 06:03:36.470196 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:36.469715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:57143)
0808 06:03:36.470035 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:36.470096 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":78,"threads_started":1}
I20260808 06:03:36.672780 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:36.672326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:34425)
0808 06:03:36.672632 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:36.672685 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":88,"threads_started":1}
I20260808 06:03:37.338137 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:37.337577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:43935)
0808 06:03:37.337982 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:37.338040 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":94,"threads_started":1}
I20260808 06:03:37.471599 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:37.471070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:36321)
0808 06:03:37.471400 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:37.471476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260808 06:03:37.673878 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:37.673474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:48061)
0808 06:03:37.673723 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:37.673790 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260808 06:03:37.979522 32043 raft_consensus.cc:482] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:37.979689 32043 raft_consensus.cc:504] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5700911098345998da25f265826fe59" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41397 } } peers { permanent_uuid: "90161dd4ad2844c68a0cda92820a1fee" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42779 } } peers { permanent_uuid: "b4c2880cb53e45b8868b8ea17cc56343" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42863 } }
I20260808 06:03:37.980182 32043 leader_election.cc:288] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5700911098345998da25f265826fe59 (127.13.229.67:41397), 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779)
W20260808 06:03:37.981019 31616 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5700911098345998da25f265826fe59 (127.13.229.67:41397): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
W20260808 06:03:37.981171 31615 leader_election.cc:341] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90161dd4ad2844c68a0cda92820a1fee (127.13.229.66:42779): Not found: Tablet not found: 041a9b5424cf4f08833e2905ce71f663
I20260808 06:03:37.981279 31615 leader_election.cc:302] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4c2880cb53e45b8868b8ea17cc56343; no voters: 90161dd4ad2844c68a0cda92820a1fee, d5700911098345998da25f265826fe59
I20260808 06:03:37.981454 32043 raft_consensus.cc:2726] T 041a9b5424cf4f08833e2905ce71f663 P b4c2880cb53e45b8868b8ea17cc56343 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 06:03:38.339277 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:38.338865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:48269)
0808 06:03:38.339124 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:38.339180 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260808 06:03:38.472934 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:38.472397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:39615)
0808 06:03:38.472701 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:38.472791 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260808 06:03:38.674974 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:38.674556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:54451)
0808 06:03:38.674831 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:38.674885 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
I20260808 06:03:39.340718 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:39.340220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:52009)
0808 06:03:39.340495 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:39.340581 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260808 06:03:39.474138 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:39.473738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:55015)
0808 06:03:39.473989 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:39.474041 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260808 06:03:39.676126 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:39.675671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:40309)
0808 06:03:39.675980 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:39.676037 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260808 06:03:40.341935 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:40.341527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:58421)
0808 06:03:40.341774 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:40.341834 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260808 06:03:40.475260 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:40.474864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:50549)
0808 06:03:40.475108 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:40.475164 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260808 06:03:40.677318 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:40.676834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:50681)
0808 06:03:40.677150 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:40.677225 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20260808 06:03:41.343114 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:41.342677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:40701)
0808 06:03:41.342956 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:41.343015 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260808 06:03:41.476488 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:41.476092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:42965)
0808 06:03:41.476332 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:41.476391 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260808 06:03:41.678805 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:41.678168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:33027)
0808 06:03:41.678534 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:41.678614 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
I20260808 06:03:42.344501 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:42.344058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:44193)
0808 06:03:42.344328 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:42.344385 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260808 06:03:42.477625 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:42.477232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:51315)
0808 06:03:42.477477 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:42.477530 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260808 06:03:42.680056 32058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:42.679570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:52383)
0808 06:03:42.679858 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:42.679961 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260808 06:03:43.345669 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:43.345243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:38449)
0808 06:03:43.345519 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:43.345573 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260808 06:03:43.478757 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:43.478356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:57887)
0808 06:03:43.478607 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:43.478662 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260808 06:03:43.681303 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:43.680854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.67:48307)
0808 06:03:43.681143 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:43.681202 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
I20260808 06:03:44.347141 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:44.346696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.65:43653)
0808 06:03:44.346992 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:44.347048 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260808 06:03:44.480293 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:44.479821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:38937 (local address 127.13.229.66:45199)
0808 06:03:44.480113 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:44.480166 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.126:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":118,"threads_started":1}
I20260808 06:03:44.664328 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31556
I20260808 06:03:44.673684 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31734
I20260808 06:03:44.680836 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 31873
2026-08-08T06:03:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59397 ms)
[----------] 30 tests from SecurityITest (241224 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:44 dist-test-slave-zkpd krb5kdc[32066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:44 dist-test-slave-zkpd krb5kdc[32066](info): set up 2 sockets
Aug 08 06:03:44 dist-test-slave-zkpd krb5kdc[32066](info): commencing operation
krb5kdc: starting...
W20260808 06:03:45.781821 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:45 dist-test-slave-zkpd krb5kdc[32066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:03:45.910305 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:32801
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:46.022408 32079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:46.022665 32079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:46.022703 32079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:46.026636 32079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:46.026695 32079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:46.026714 32079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:46.026731 32079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:46.026746 32079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:46.031808 32079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:46.032869 32079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:46.033881 32079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:46.042979 32085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.043009 32087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.042979 32084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:46.043285 32079 server_base.cc:1034] running on GCE node
I20260808 06:03:46.043833 32079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:46.045010 32079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:46.046221 32079 hybrid_clock.cc:648] HybridClock initialized: now 1786169026046189 us; error 47 us; skew 500 ppm
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:46.049057 32079 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:03:46.049868 32079 webserver.cc:460] Webserver started at http://127.13.229.126:33787/ using document root <none> and password file <none>
I20260808 06:03:46.050390 32079 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:46.050451 32079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:46.050622 32079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:46.052706 32079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "523404f46548475cbe89c90a83f86553"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f417c82f17031b5dc94bb0f595036f4d"
server_key_iv: "52bee3f4b01da8ca87a54430b568101e"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.053231 32079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "523404f46548475cbe89c90a83f86553"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f417c82f17031b5dc94bb0f595036f4d"
server_key_iv: "52bee3f4b01da8ca87a54430b568101e"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.056854 32079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:03:46.059116 32094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.060411 32079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:03:46.060513 32079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "523404f46548475cbe89c90a83f86553"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f417c82f17031b5dc94bb0f595036f4d"
server_key_iv: "52bee3f4b01da8ca87a54430b568101e"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.060596 32079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:46.081171 32079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:46.084081 32079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:46.084254 32079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:46.091368 32079 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:32801
I20260808 06:03:46.091393 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:32801 every 8 connection(s)
I20260808 06:03:46.092374 32079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:46.095155 32152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:46.097276 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32079
I20260808 06:03:46.097420 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:46.097699 14229 external_mini_cluster.cc:1428] Setting key de3de2053d293177e3619adfbf294567
I20260808 06:03:46.102447 32152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: Bootstrap starting.
I20260808 06:03:46.104893 32152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:46.105777 32152 log.cc:824] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:46.107960 32152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: No bootstrap required, opened a new log
I20260808 06:03:46.111616 32152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } }
I20260808 06:03:46.111835 32152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:46.111871 32152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523404f46548475cbe89c90a83f86553, State: Initialized, Role: FOLLOWER
I20260808 06:03:46.112319 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.099145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.13.229.126:32801)
0808 06:03:46.099576 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.099596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.099640 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:03:46.100413 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.100437 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.100857 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.101245 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:46.102032 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.103257 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.104188 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.104662 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.104839 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.107625 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:46.107630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:46.107644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:46.107671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:46.109353 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.109817 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.109822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.109827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.109938 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.110228 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.110232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.110233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.110631 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:46.110853 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.111169 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.111461 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260808 06:03:46.112378 32152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } }
I20260808 06:03:46.112551 32152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:46.112608 32152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:46.112690 32152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:46.113628 32152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } }
I20260808 06:03:46.114001 32152 leader_election.cc:302] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523404f46548475cbe89c90a83f86553; no voters:
I20260808 06:03:46.114332 32152 leader_election.cc:288] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:46.114444 32158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:46.114670 32158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [term 1 LEADER]: Becoming Leader. State: Replica: 523404f46548475cbe89c90a83f86553, State: Running, Role: LEADER
I20260808 06:03:46.115012 32158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } }
I20260808 06:03:46.115573 32152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:46.116870 32160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523404f46548475cbe89c90a83f86553. Latest consensus state: current_term: 1 leader_uuid: "523404f46548475cbe89c90a83f86553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } } }
I20260808 06:03:46.116876 32159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523404f46548475cbe89c90a83f86553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523404f46548475cbe89c90a83f86553" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32801 } } }
I20260808 06:03:46.117056 32160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:46.117094 32159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:46.117497 32168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:46.120662 32168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:46.125959 32168 catalog_manager.cc:1347] Generated new cluster ID: 45f65adaebac4962ae59cac20cdb169d
I20260808 06:03:46.126029 32168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:46.145568 32168 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:46.146535 32168 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:46.159591 32168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: Generated new TSK 0
I20260808 06:03:46.160444 32168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:03:46.190989 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:46.306249 32179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:46.306552 32179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:46.306612 32179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:46.310557 32179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:46.310607 32179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:46.310662 32179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:46.315610 32179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:46.316778 32179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:46.317804 32179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:46.326922 32184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.326931 32187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.327035 32185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:46.327575 32179 server_base.cc:1034] running on GCE node
I20260808 06:03:46.327996 32179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:46.328595 32179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:46.329784 32179 hybrid_clock.cc:648] HybridClock initialized: now 1786169026329763 us; error 28 us; skew 500 ppm
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:46.332489 32179 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:03:46.333300 32179 webserver.cc:460] Webserver started at http://127.13.229.65:37995/ using document root <none> and password file <none>
I20260808 06:03:46.333853 32179 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:46.333943 32179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:46.334173 32179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:46.336215 32179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "85d2206c45fa4ec0af1cd57ad2e929c3"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8b17fffb812891baf943e9ccfa8e4a06"
server_key_iv: "e163f579cde04df3314c6139e0bbc432"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.336781 32179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "85d2206c45fa4ec0af1cd57ad2e929c3"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8b17fffb812891baf943e9ccfa8e4a06"
server_key_iv: "e163f579cde04df3314c6139e0bbc432"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.340356 32179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:03:46.342605 32194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.343763 32179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:46.343956 32179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "85d2206c45fa4ec0af1cd57ad2e929c3"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8b17fffb812891baf943e9ccfa8e4a06"
server_key_iv: "e163f579cde04df3314c6139e0bbc432"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.344069 32179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:46.364297 32179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:46.367172 32179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:46.367400 32179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:46.368080 32179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:46.368939 32179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:46.369016 32179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.369091 32179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:46.369138 32179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.378504 32179 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:43177
I20260808 06:03:46.378537 32315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:43177 every 8 connection(s)
I20260808 06:03:46.379456 32179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:46.387840 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32179
I20260808 06:03:46.388007 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:46.388263 14229 external_mini_cluster.cc:1428] Setting key a13dd5d1ab02bb90d369c3e6d0a4602c
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:46.394322 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.381449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:45893 (local address 127.13.229.126:32801)
0808 06:03:46.381590 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.381594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.382928 (+ 1334us) server_negotiation.cc:407] Connection header received
0808 06:03:46.383867 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.383870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.383957 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.384039 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:46.385298 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.385856 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.387090 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.387339 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.387457 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.390242 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:46.390249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:46.390254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:46.390298 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:46.391971 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.392631 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.392633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.392634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.392710 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.393103 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.393105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.393106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.393294 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:46.393371 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.394014 (+ 643us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.394144 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 06:03:46.395184 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.381833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32801 (local address 127.13.229.65:45893)
0808 06:03:46.382755 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.382814 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.383654 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.384216 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.384224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.384648 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:46.385142 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.385157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.385999 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.386008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.386958 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.386969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.387503 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.387507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.387684 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.388379 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:46.388396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:46.390066 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:46.392103 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.392112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.392131 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.392521 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.392798 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.392803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.392805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.393005 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.393399 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:46.393408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:46.393746 (+ 338us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.393950 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.394284 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":566,"threads_started":1}
I20260808 06:03:46.396484 32316 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32801
I20260808 06:03:46.396724 32316 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:46.397226 32316 heartbeater.cc:511] Master 127.13.229.126:32801 requested a full tablet report, sending...
I20260808 06:03:46.398710 32112 ts_manager.cc:194] Registered new tserver with Master: 85d2206c45fa4ec0af1cd57ad2e929c3 (127.13.229.65:43177)
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:03:46.399884 32112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:45893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:03:46.418434 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:46.532222 32321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:46.532464 32321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:46.532505 32321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:46.536547 32321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:46.536603 32321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:46.536659 32321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:46.541762 32321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:46.542704 32321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:46.543642 32321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:46.553248 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.553256 32326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.553256 32329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:46.553414 32321 server_base.cc:1034] running on GCE node
I20260808 06:03:46.553949 32321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:46.554795 32321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:46.555967 32321 hybrid_clock.cc:648] HybridClock initialized: now 1786169026555950 us; error 32 us; skew 500 ppm
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:46.558590 32321 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:03:46.559468 32321 webserver.cc:460] Webserver started at http://127.13.229.66:35239/ using document root <none> and password file <none>
I20260808 06:03:46.560170 32321 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:46.560266 32321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:46.560477 32321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:46.562429 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "bbb210b6a0ee4374bb41779ca8932a89"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8ce087fe86be836f22d9a9bed49634e0"
server_key_iv: "775be6d3dd132858a95d1cf212bbc462"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.563001 32321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "bbb210b6a0ee4374bb41779ca8932a89"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8ce087fe86be836f22d9a9bed49634e0"
server_key_iv: "775be6d3dd132858a95d1cf212bbc462"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.566674 32321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:03:46.569036 32336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.570176 32321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:46.570312 32321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "bbb210b6a0ee4374bb41779ca8932a89"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "8ce087fe86be836f22d9a9bed49634e0"
server_key_iv: "775be6d3dd132858a95d1cf212bbc462"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.570420 32321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:46.585992 32321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:46.588888 32321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:46.589094 32321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:46.589663 32321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:46.590479 32321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:46.590551 32321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.590623 32321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:46.590667 32321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.599725 32321 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:33811
I20260808 06:03:46.599752 32454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:33811 every 8 connection(s)
I20260808 06:03:46.600683 32321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:46.605474 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32321
I20260808 06:03:46.605629 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:46.605962 14229 external_mini_cluster.cc:1428] Setting key a6caadd4ac94a94508f38394febc1eca
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:46.615664 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.602500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:37205 (local address 127.13.229.126:32801)
0808 06:03:46.602648 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.602652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.603486 (+ 834us) server_negotiation.cc:407] Connection header received
0808 06:03:46.604954 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.604958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.605019 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.605122 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:46.606650 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.607380 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.608586 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.608811 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.608934 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.611569 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:46.611576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:46.611582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:46.611616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:46.613334 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.614019 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.614022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.614023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.614100 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.614519 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.614521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.614522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.614693 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:46.614766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.615422 (+ 656us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.615516 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:03:46.616978 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.602857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32801 (local address 127.13.229.66:37205)
0808 06:03:46.603328 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.603370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.604730 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.605327 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.605336 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.605841 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:46.606472 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.606488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.607519 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.607522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.608457 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.608468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.608967 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.608971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.609137 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.609728 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:46.609744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:46.611341 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:46.613478 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.613487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.613504 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.613914 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.614201 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.614206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.614210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.614412 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.614805 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:46.614813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:46.615164 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.615387 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.615767 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:03:46.618327 32456 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32801
I20260808 06:03:46.618602 32456 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:46.619134 32456 heartbeater.cc:511] Master 127.13.229.126:32801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:03:46.620267 32112 ts_manager.cc:194] Registered new tserver with Master: bbb210b6a0ee4374bb41779ca8932a89 (127.13.229.66:33811)
I20260808 06:03:46.620882 32112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:37205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:03:46.636252 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:46.747984 32461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:46.748242 32461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:46.748309 32461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:46.752485 32461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:46.752535 32461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:46.752590 32461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:46.757596 32461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:46.758473 32461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:46.759389 32461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:46.768848 32467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.768834 32466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:46.768833 32469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:46.769173 32461 server_base.cc:1034] running on GCE node
I20260808 06:03:46.769631 32461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:46.770231 32461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:46.771425 32461 hybrid_clock.cc:648] HybridClock initialized: now 1786169026771414 us; error 31 us; skew 500 ppm
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:46.774634 32461 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:03:46.775486 32461 webserver.cc:460] Webserver started at http://127.13.229.67:34971/ using document root <none> and password file <none>
I20260808 06:03:46.776103 32461 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:46.776167 32461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:46.776418 32461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:46.778427 32461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "d455304b2e5142a7837f294158ca91ec"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f8a12aa715ddd3e9e41e1e0bc1db06a5"
server_key_iv: "242b855cd53ed8720bc5cfbf5202ce76"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.778990 32461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "d455304b2e5142a7837f294158ca91ec"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f8a12aa715ddd3e9e41e1e0bc1db06a5"
server_key_iv: "242b855cd53ed8720bc5cfbf5202ce76"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.782780 32461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 06:03:46.785113 32476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.786401 32461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:46.786540 32461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "d455304b2e5142a7837f294158ca91ec"
format_stamp: "Formatted at 2026-08-08 06:03:46 on dist-test-slave-zkpd"
server_key: "f8a12aa715ddd3e9e41e1e0bc1db06a5"
server_key_iv: "242b855cd53ed8720bc5cfbf5202ce76"
server_key_version: "encryptionkey@0"
I20260808 06:03:46.786643 32461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:46.813079 32461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:46.816073 32461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:46.816274 32461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:46.817003 32461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:46.817888 32461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:46.817943 32461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.818009 32461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:46.818046 32461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:46.826872 32461 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41781
I20260808 06:03:46.826910 32592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41781 every 8 connection(s)
I20260808 06:03:46.827788 32461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:46.833244 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32461
I20260808 06:03:46.833413 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:46.833693 14229 external_mini_cluster.cc:1428] Setting key d28b008d3ff7f9c3ce343421ebf12c8f
Aug 08 06:03:46 dist-test-slave-zkpd krb5kdc[32066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169026, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:46.841401 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.829633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:52367 (local address 127.13.229.126:32801)
0808 06:03:46.829784 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.829788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.830584 (+ 796us) server_negotiation.cc:407] Connection header received
0808 06:03:46.831962 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.831966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.832027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.832134 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:46.833296 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.833984 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.835264 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.835566 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.835674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.838131 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:46.838134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:46.838136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:46.838160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:46.839584 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.840149 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.840151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.840152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.840225 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.840516 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.840518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.840519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.840678 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:46.840752 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.841191 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.841282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:03:46.842203 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.829993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32801 (local address 127.13.229.67:52367)
0808 06:03:46.830429 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.830471 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.831721 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.832309 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.832315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.832666 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:46.833152 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.833166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.834148 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.834151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.835155 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.835166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.835729 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.835733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.835955 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.836490 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:46.836506 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:46.837975 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:46.839706 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.839714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.839730 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.840062 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.840306 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:46.840308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:46.840309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:46.840437 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:46.840764 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:46.840769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:46.840995 (+ 226us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.841144 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.841378 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260808 06:03:46.843272 32593 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32801
I20260808 06:03:46.843519 32593 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:46.844043 32593 heartbeater.cc:511] Master 127.13.229.126:32801 requested a full tablet report, sending...
I20260808 06:03:46.845022 32112 ts_manager.cc:194] Registered new tserver with Master: d455304b2e5142a7837f294158ca91ec (127.13.229.67:41781)
I20260808 06:03:46.845577 32112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:52367
I20260808 06:03:46.847472 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:46.856180 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.849570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.13.229.126:32801)
0808 06:03:46.849680 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.849683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.849845 (+ 162us) server_negotiation.cc:407] Connection header received
0808 06:03:46.849946 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.849948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.849980 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.850043 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:46.850647 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.851144 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.852047 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.852262 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.852339 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.853594 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:46.853596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:46.853598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:46.853615 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:46.854763 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.855161 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.855163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.855164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.855232 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:46.855456 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:46.855458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:46.855459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:46.855642 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:46.855741 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.855992 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.856079 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:03:46.858659 32112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:46.861647 32112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:46.876384 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.869792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41781 (local address 127.0.0.1:39248)
0808 06:03:46.870814 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.870829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.870998 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.871471 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.871474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.871493 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:46.871692 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.871700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.873268 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.873273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.874600 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.874612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875862 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.875977 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.876001 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.876110 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.876234 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":68,"thread_start_us":102,"threads_started":1}
I20260808 06:03:46.876466 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.870426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33811 (local address 127.0.0.1:49870)
0808 06:03:46.871222 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.871235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.871316 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.871799 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.871804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.871826 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:46.871958 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.871965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.873307 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.873310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.874714 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.874724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875984 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.876121 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.876155 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.876229 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.876301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":226,"thread_start_us":92,"threads_started":1}
I20260808 06:03:46.876785 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.869854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.13.229.65:43177)
0808 06:03:46.870776 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.870792 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.870816 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:03:46.870894 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.870900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.871109 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.871263 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:46.871861 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.872699 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.874255 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875117 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875460 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.875981 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.876187 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.876644 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":67,"threads_started":1}
I20260808 06:03:46.876971 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.870206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.13.229.67:41781)
0808 06:03:46.870672 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.870685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.870925 (+ 240us) server_negotiation.cc:407] Connection header received
0808 06:03:46.871164 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.871172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.871337 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.871482 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:46.871840 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.873079 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.874722 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875562 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875665 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.875800 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.876718 (+ 918us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.876817 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
I20260808 06:03:46.877211 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.869716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:43177 (local address 127.0.0.1:43074)
0808 06:03:46.870293 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.870361 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.870494 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.871446 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.871452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.871476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:46.871709 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.871716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.872832 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.872839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.874144 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.874156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875409 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.875764 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.875941 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.876110 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.876364 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":55,"spinlock_wait_cycles":896,"thread_start_us":67,"threads_started":1}
I20260808 06:03:46.877313 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.870750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.13.229.66:33811)
0808 06:03:46.871315 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.871326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.871349 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:03:46.871452 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.871457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.871642 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.871814 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:46.872058 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.873140 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.874841 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.875595 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.875722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.875911 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.877067 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.877174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":125,"threads_started":1}
I20260808 06:03:46.879623 32525 tablet_service.cc:1467] Processing CreateTablet for tablet 293f3d95996547ccb11046260cbe2118 (DEFAULT_TABLE table=test-table [id=3ccaaac89e33441e88560af0892ba959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:46.879624 32246 tablet_service.cc:1467] Processing CreateTablet for tablet 293f3d95996547ccb11046260cbe2118 (DEFAULT_TABLE table=test-table [id=3ccaaac89e33441e88560af0892ba959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:46.880752 32525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293f3d95996547ccb11046260cbe2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:46.880769 32246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293f3d95996547ccb11046260cbe2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:46.881738 32386 tablet_service.cc:1467] Processing CreateTablet for tablet 293f3d95996547ccb11046260cbe2118 (DEFAULT_TABLE table=test-table [id=3ccaaac89e33441e88560af0892ba959]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:46.882670 32386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293f3d95996547ccb11046260cbe2118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:46.887018 32611 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Bootstrap starting.
I20260808 06:03:46.888113 32612 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: Bootstrap starting.
I20260808 06:03:46.889531 32611 tablet_bootstrap.cc:654] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:46.889824 32613 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: Bootstrap starting.
I20260808 06:03:46.890434 32612 tablet_bootstrap.cc:654] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:46.891201 32611 log.cc:824] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:46.891839 32612 log.cc:824] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:46.891947 32613 tablet_bootstrap.cc:654] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:46.893190 32613 log.cc:824] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:46.893443 32611 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: No bootstrap required, opened a new log
I20260808 06:03:46.893591 32612 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: No bootstrap required, opened a new log
I20260808 06:03:46.893648 32611 ts_tablet_manager.cc:1397] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 06:03:46.893802 32612 ts_tablet_manager.cc:1397] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:46.895085 32613 tablet_bootstrap.cc:492] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: No bootstrap required, opened a new log
I20260808 06:03:46.895295 32613 ts_tablet_manager.cc:1397] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 06:03:46.896757 32612 raft_consensus.cc:348] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.897002 32612 raft_consensus.cc:374] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:46.897065 32612 raft_consensus.cc:729] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d2206c45fa4ec0af1cd57ad2e929c3, State: Initialized, Role: FOLLOWER
I20260808 06:03:46.897576 32612 consensus_queue.cc:260] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.897571 32611 raft_consensus.cc:348] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.897799 32611 raft_consensus.cc:374] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:46.897853 32611 raft_consensus.cc:729] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d455304b2e5142a7837f294158ca91ec, State: Initialized, Role: FOLLOWER
I20260808 06:03:46.898319 32611 consensus_queue.cc:260] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.898597 32612 ts_tablet_manager.cc:1428] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:03:46.898758 32316 heartbeater.cc:503] Master 127.13.229.126:32801 was elected leader, sending a full tablet report...
I20260808 06:03:46.899078 32593 heartbeater.cc:503] Master 127.13.229.126:32801 was elected leader, sending a full tablet report...
I20260808 06:03:46.899228 32611 ts_tablet_manager.cc:1428] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:03:46.900565 32613 raft_consensus.cc:348] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.900810 32613 raft_consensus.cc:374] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:46.900893 32613 raft_consensus.cc:729] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb210b6a0ee4374bb41779ca8932a89, State: Initialized, Role: FOLLOWER
I20260808 06:03:46.901288 32613 consensus_queue.cc:260] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.901880 32456 heartbeater.cc:503] Master 127.13.229.126:32801 was elected leader, sending a full tablet report...
I20260808 06:03:46.902169 32613 ts_tablet_manager.cc:1428] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20260808 06:03:46.964702 32618 raft_consensus.cc:482] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:46.964970 32618 raft_consensus.cc:504] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.966251 32618 leader_election.cc:288] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85d2206c45fa4ec0af1cd57ad2e929c3 (127.13.229.65:43177), bbb210b6a0ee4374bb41779ca8932a89 (127.13.229.66:33811)
I20260808 06:03:46.970307 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.966599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:45225 (local address 127.13.229.65:43177)
0808 06:03:46.966731 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.966736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.966752 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:03:46.966815 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.966819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.966886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.966983 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:46.967376 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.968105 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969299 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.969715 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969808 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.969895 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.970093 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.970173 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:03:46.970301 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.966441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:43177 (local address 127.13.229.67:45225)
0808 06:03:46.966577 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.966594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.966689 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.967063 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.967067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.967088 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:46.967224 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.967231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.968211 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.968214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.969160 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.969170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.969836 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.969949 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.969982 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.970060 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.970155 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":48}
I20260808 06:03:46.970335 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.966850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:50505 (local address 127.13.229.66:33811)
0808 06:03:46.967027 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.967033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.967231 (+ 198us) server_negotiation.cc:407] Connection header received
0808 06:03:46.967315 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.967318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.967378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.967439 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:46.967721 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.968310 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969352 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.969753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969892 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.969950 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.970150 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.970224 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 06:03:46.970320 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.966722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33811 (local address 127.13.229.67:50505)
0808 06:03:46.967125 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:46.967138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:46.967232 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:46.967471 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:46.967474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:46.967491 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:46.967611 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.967617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.968420 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.968423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.969196 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:46.969204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.969901 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.969907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:46.970001 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.970020 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 06:03:46.970116 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 06:03:46.970182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260808 06:03:46.970997 32266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293f3d95996547ccb11046260cbe2118" candidate_uuid: "d455304b2e5142a7837f294158ca91ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" is_pre_election: true
I20260808 06:03:46.971342 32266 raft_consensus.cc:2454] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d455304b2e5142a7837f294158ca91ec in term 0.
I20260808 06:03:46.971403 32407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293f3d95996547ccb11046260cbe2118" candidate_uuid: "d455304b2e5142a7837f294158ca91ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb210b6a0ee4374bb41779ca8932a89" is_pre_election: true
I20260808 06:03:46.971690 32407 raft_consensus.cc:2454] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d455304b2e5142a7837f294158ca91ec in term 0.
I20260808 06:03:46.971912 32479 leader_election.cc:302] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d2206c45fa4ec0af1cd57ad2e929c3, d455304b2e5142a7837f294158ca91ec; no voters:
I20260808 06:03:46.972191 32618 raft_consensus.cc:2781] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:46.972285 32618 raft_consensus.cc:482] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:46.972329 32618 raft_consensus.cc:3037] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:46.973454 32618 raft_consensus.cc:504] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.973805 32618 leader_election.cc:288] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [CANDIDATE]: Term 1 election: Requested vote from peers 85d2206c45fa4ec0af1cd57ad2e929c3 (127.13.229.65:43177), bbb210b6a0ee4374bb41779ca8932a89 (127.13.229.66:33811)
I20260808 06:03:46.974159 32407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293f3d95996547ccb11046260cbe2118" candidate_uuid: "d455304b2e5142a7837f294158ca91ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb210b6a0ee4374bb41779ca8932a89"
I20260808 06:03:46.974159 32266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293f3d95996547ccb11046260cbe2118" candidate_uuid: "d455304b2e5142a7837f294158ca91ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3"
I20260808 06:03:46.974282 32407 raft_consensus.cc:3037] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:46.974282 32266 raft_consensus.cc:3037] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:46.975600 32266 raft_consensus.cc:2454] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d455304b2e5142a7837f294158ca91ec in term 1.
I20260808 06:03:46.975600 32407 raft_consensus.cc:2454] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d455304b2e5142a7837f294158ca91ec in term 1.
I20260808 06:03:46.975970 32479 leader_election.cc:302] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d2206c45fa4ec0af1cd57ad2e929c3, d455304b2e5142a7837f294158ca91ec; no voters:
I20260808 06:03:46.976137 32618 raft_consensus.cc:2781] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:46.976352 32618 raft_consensus.cc:686] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 1 LEADER]: Becoming Leader. State: Replica: d455304b2e5142a7837f294158ca91ec, State: Running, Role: LEADER
I20260808 06:03:46.976677 32618 consensus_queue.cc:237] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } }
I20260808 06:03:46.979830 32110 catalog_manager.cc:5495] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec reported cstate change: term changed from 0 to 1, leader changed from <none> to d455304b2e5142a7837f294158ca91ec (127.13.229.67). New cstate: current_term: 1 leader_uuid: "d455304b2e5142a7837f294158ca91ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d455304b2e5142a7837f294158ca91ec" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 } health_report { overall_health: UNKNOWN } } }
I20260808 06:03:46.996068 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:46.992824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.13.229.67:41781)
0808 06:03:46.992955 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:46.992958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:46.992970 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:03:46.993121 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:46.993123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:46.993176 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:46.993260 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:46.993470 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.993962 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.994758 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:46.994969 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:46.995045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:46.995298 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:46.995689 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:46.995779 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:46.995834 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 06:03:46.995932 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:03:47.001734 32407 raft_consensus.cc:1261] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 1 FOLLOWER]: Refusing update from remote peer d455304b2e5142a7837f294158ca91ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:47.001748 32266 raft_consensus.cc:1261] T 293f3d95996547ccb11046260cbe2118 P 85d2206c45fa4ec0af1cd57ad2e929c3 [term 1 FOLLOWER]: Refusing update from remote peer d455304b2e5142a7837f294158ca91ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:47.002440 32618 consensus_queue.cc:1035] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d2206c45fa4ec0af1cd57ad2e929c3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 43177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:47.002620 32621 consensus_queue.cc:1035] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb210b6a0ee4374bb41779ca8932a89" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:47.012205 32624 mvcc.cc:205] Tried to move back new op lower bound from 7316148334592315392 to 7316148334499291136. Current Snapshot: MvccSnapshot[applied={T|T < 7316148334592315392 or (T in {7316148334592315392})}]
I20260808 06:03:47.012413 32626 mvcc.cc:205] Tried to move back new op lower bound from 7316148334592315392 to 7316148334499291136. Current Snapshot: MvccSnapshot[applied={T|T < 7316148334592315392}]
I20260808 06:03:47.013307 32625 mvcc.cc:205] Tried to move back new op lower bound from 7316148334592315392 to 7316148334499291136. Current Snapshot: MvccSnapshot[applied={T|T < 7316148334592315392}]
I20260808 06:03:47.017948 32110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:47.018114 32110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:47.020582 32110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523404f46548475cbe89c90a83f86553: Sending DeleteTablet for 3 replicas of tablet 293f3d95996547ccb11046260cbe2118
I20260808 06:03:47.021418 32246 tablet_service.cc:1514] Processing DeleteTablet for tablet 293f3d95996547ccb11046260cbe2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:43074
I20260808 06:03:47.021399 32386 tablet_service.cc:1514] Processing DeleteTablet for tablet 293f3d95996547ccb11046260cbe2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:49870
I20260808 06:03:47.021450 32525 tablet_service.cc:1514] Processing DeleteTablet for tablet 293f3d95996547ccb11046260cbe2118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:47 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:39248
I20260808 06:03:47.021876 32633 tablet_replica.cc:330] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89: stopping tablet replica
I20260808 06:03:47.022039 32635 tablet_replica.cc:330] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: stopping tablet replica
I20260808 06:03:47.022311 32635 raft_consensus.cc:2229] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:47.022536 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32179
I20260808 06:03:47.022708 32635 raft_consensus.cc:2258] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:47.024811 32635 ts_tablet_manager.cc:1905] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:47.025169 32633 raft_consensus.cc:2229] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:47.025529 32633 raft_consensus.cc:2258] T 293f3d95996547ccb11046260cbe2118 P bbb210b6a0ee4374bb41779ca8932a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:47.027666 32635 ts_tablet_manager.cc:1918] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:47.027771 32635 log.cc:1196] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/293f3d95996547ccb11046260cbe2118
I20260808 06:03:47.028163 32635 ts_tablet_manager.cc:1939] T 293f3d95996547ccb11046260cbe2118 P d455304b2e5142a7837f294158ca91ec: Deleting consensus metadata
I20260808 06:03:47.029371 32095 catalog_manager.cc:4841] TS d455304b2e5142a7837f294158ca91ec (127.13.229.67:41781): tablet 293f3d95996547ccb11046260cbe2118 (table test-table [id=3ccaaac89e33441e88560af0892ba959]) successfully deleted
W20260808 06:03:47.032502 32097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:43177 (error 108)
I20260808 06:03:47.032618 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32321
I20260808 06:03:47.034991 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.034622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:43177 (local address 127.0.0.1:43076)
0808 06:03:47.034791 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.034883 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:43177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 06:03:47.035177 32097 catalog_manager.cc:4584] TS 85d2206c45fa4ec0af1cd57ad2e929c3 (127.13.229.65:43177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 293f3d95996547ccb11046260cbe2118: Network error: Client connection negotiation failed: client connection to 127.13.229.65:43177: connect: Connection refused (error 111)
I20260808 06:03:47.039870 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32461
I20260808 06:03:47.041975 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.040064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:33811 (local address 127.0.0.1:49880)
0808 06:03:47.041793 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.041863 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.66:33811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1626}
I20260808 06:03:47.046710 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32079
2026-08-08T06:03:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2365 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T06:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:47Z Disabled control of system clock
I20260808 06:03:47.080309 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36131
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:36131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:47.191879 32640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:47.192138 32640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:47.192173 32640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:47.196079 32640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:47.196128 32640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:47.196143 32640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:47.196161 32640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:47.196175 32640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:47.200726 32640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:36131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:47.201701 32640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:47.202994 32640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:47.211174 32648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:47.211257 32640 server_base.cc:1034] running on GCE node
W20260808 06:03:47.211159 32646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.211177 32645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:47.212038 32640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:47.213063 32640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:47.214222 32640 hybrid_clock.cc:648] HybridClock initialized: now 1786169027214195 us; error 39 us; skew 500 ppm
I20260808 06:03:47.216040 32640 webserver.cc:460] Webserver started at http://127.13.229.126:39729/ using document root <none> and password file <none>
I20260808 06:03:47.216621 32640 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:47.216682 32640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:47.216922 32640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:47.218907 32640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "5a37ee68a01e46598cd0af14e90d6237"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "bf68cea9dc864dee5f9040e0176ea661"
server_key_iv: "588f0436b4e89a50a75e92ee606097f3"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.219465 32640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "5a37ee68a01e46598cd0af14e90d6237"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "bf68cea9dc864dee5f9040e0176ea661"
server_key_iv: "588f0436b4e89a50a75e92ee606097f3"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.223016 32640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:03:47.225419 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.226644 32640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:47.226742 32640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "5a37ee68a01e46598cd0af14e90d6237"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "bf68cea9dc864dee5f9040e0176ea661"
server_key_iv: "588f0436b4e89a50a75e92ee606097f3"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.226819 32640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:47.240690 32640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:47.241264 32640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:47.241408 32640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:47.248453 32640 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:36131
I20260808 06:03:47.248471 32707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:36131 every 8 connection(s)
I20260808 06:03:47.249354 32640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:47.252439 32708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:47.255735 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32640
I20260808 06:03:47.255935 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:47.256232 14229 external_mini_cluster.cc:1428] Setting key 9542e483f6ac67c475ba6aca3d448c4b
I20260808 06:03:47.258387 32708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: Bootstrap starting.
I20260808 06:03:47.260740 32708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:47.261644 32708 log.cc:824] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:47.263942 32708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: No bootstrap required, opened a new log
I20260808 06:03:47.265594 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.257554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.13.229.126:36131)
0808 06:03:47.258141 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.258158 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.258193 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:03:47.258904 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.258924 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.259313 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.259588 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:47.259995 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.261016 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.262214 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.262779 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.262904 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.263752 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:47.263757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:47.263769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:47.264260 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:47.264315 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:47.264336 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:47.264416 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.264599 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.264910 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
I20260808 06:03:47.267664 32708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } }
I20260808 06:03:47.267979 32708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:47.268051 32708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a37ee68a01e46598cd0af14e90d6237, State: Initialized, Role: FOLLOWER
I20260808 06:03:47.268477 32708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } }
I20260808 06:03:47.268620 32708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:47.268687 32708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:47.268807 32708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:47.269699 32708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } }
I20260808 06:03:47.270051 32708 leader_election.cc:302] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a37ee68a01e46598cd0af14e90d6237; no voters:
I20260808 06:03:47.270349 32708 leader_election.cc:288] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:47.270489 32713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:47.270736 32713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [term 1 LEADER]: Becoming Leader. State: Replica: 5a37ee68a01e46598cd0af14e90d6237, State: Running, Role: LEADER
I20260808 06:03:47.271066 32713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } }
I20260808 06:03:47.271577 32708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:47.272733 32715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a37ee68a01e46598cd0af14e90d6237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } } }
I20260808 06:03:47.272864 32715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:47.273303 32724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:47.273664 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a37ee68a01e46598cd0af14e90d6237. Latest consensus state: current_term: 1 leader_uuid: "5a37ee68a01e46598cd0af14e90d6237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a37ee68a01e46598cd0af14e90d6237" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36131 } } }
I20260808 06:03:47.273783 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:47.276600 32724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:47.281679 32724 catalog_manager.cc:1347] Generated new cluster ID: f08d2d708ab945f18366e5e12aef9b91
I20260808 06:03:47.281783 32724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:47.307680 32724 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:47.308691 32724 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:47.315209 32724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: Generated new TSK 0
I20260808 06:03:47.315944 32724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:47.326283 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--builtin_ntp_servers=127.13.229.84:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:47.455437 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:47.455677 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:47.455713 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:47.459726 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:47.459810 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:47.459950 32734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:47.465306 32734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:47.466536 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:47.467998 32734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:47.477437 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.477437 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.477447 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:47.477797 32734 server_base.cc:1034] running on GCE node
I20260808 06:03:47.478207 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:47.478768 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:47.479965 32734 hybrid_clock.cc:648] HybridClock initialized: now 1786169027479942 us; error 33 us; skew 500 ppm
I20260808 06:03:47.481709 32734 webserver.cc:460] Webserver started at http://127.13.229.65:37003/ using document root <none> and password file <none>
I20260808 06:03:47.482267 32734 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:47.482328 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:47.482484 32734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:47.484542 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "f14268b42bad4b3b9305b648154cb476"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "0558d8ea97eeff8f6a466a969192c951"
server_key_iv: "d0753f80fd8d9891de73614f37064bb7"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.485057 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "f14268b42bad4b3b9305b648154cb476"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "0558d8ea97eeff8f6a466a969192c951"
server_key_iv: "d0753f80fd8d9891de73614f37064bb7"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.488718 32734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:03:47.490998 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.492221 32734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:47.492338 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "f14268b42bad4b3b9305b648154cb476"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "0558d8ea97eeff8f6a466a969192c951"
server_key_iv: "d0753f80fd8d9891de73614f37064bb7"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.492441 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:47.505828 32734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:47.506381 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:47.506518 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:47.507009 32734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:47.507831 32734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:47.507885 32734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.507978 32734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:47.508013 32734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.517040 32734 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:39321
I20260808 06:03:47.517074 397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:39321 every 8 connection(s)
I20260808 06:03:47.517865 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:47.524313 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 32734
I20260808 06:03:47.524467 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:47.524755 14229 external_mini_cluster.cc:1428] Setting key 2f72f2c0bdc4d5a5406c40bcbbb8e37b
I20260808 06:03:47.526971 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--builtin_ntp_servers=127.13.229.84:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:47.527467 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.519642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40041 (local address 127.13.229.126:36131)
0808 06:03:47.519789 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.519793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.520888 (+ 1095us) server_negotiation.cc:407] Connection header received
0808 06:03:47.522327 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.522330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.522399 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.522490 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:47.523370 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.523980 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.525270 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.525542 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.525649 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.526844 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:47.526847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:47.526849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:47.526861 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:47.526880 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:47.526889 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:47.526948 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.527181 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.527254 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:03:47.529021 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.520083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36131 (local address 127.13.229.65:40041)
0808 06:03:47.520708 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.520782 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.522098 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.522661 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.522668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.522753 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:47.523227 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.523241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.524106 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.524119 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.525141 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.525151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.525691 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.525694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.525842 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.526516 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:47.526518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:47.526582 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:47.526681 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:47.526981 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:47.526993 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:47.527000 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.527657 (+ 657us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.527988 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260808 06:03:47.530225 398 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36131
I20260808 06:03:47.530434 398 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:47.530932 398 heartbeater.cc:511] Master 127.13.229.126:36131 requested a full tablet report, sending...
I20260808 06:03:47.532428 32671 ts_manager.cc:194] Registered new tserver with Master: f14268b42bad4b3b9305b648154cb476 (127.13.229.65:39321)
I20260808 06:03:47.533829 32671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.65:40041
W20260808 06:03:47.645954 401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:47.646245 401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:47.646330 401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:47.651154 401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:47.651230 401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:47.651315 401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:47.656170 401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:47.657249 401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:47.658612 401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:47.668361 407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.668365 406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.668365 409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:47.668447 401 server_base.cc:1034] running on GCE node
I20260808 06:03:47.668964 401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:47.669574 401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:47.670761 401 hybrid_clock.cc:648] HybridClock initialized: now 1786169027670748 us; error 25 us; skew 500 ppm
I20260808 06:03:47.672660 401 webserver.cc:460] Webserver started at http://127.13.229.66:34977/ using document root <none> and password file <none>
I20260808 06:03:47.673264 401 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:47.673357 401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:47.673584 401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:47.675660 401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "7e25633bfa22497092e48a6925115bfb"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "97c23da788aa8a8889abb4360e7513ce"
server_key_iv: "5a0ba57207a4aea432c2fe751ecbdff0"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.676318 401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "7e25633bfa22497092e48a6925115bfb"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "97c23da788aa8a8889abb4360e7513ce"
server_key_iv: "5a0ba57207a4aea432c2fe751ecbdff0"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.679765 401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:03:47.682071 415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.683174 401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:47.683307 401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "7e25633bfa22497092e48a6925115bfb"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "97c23da788aa8a8889abb4360e7513ce"
server_key_iv: "5a0ba57207a4aea432c2fe751ecbdff0"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.683409 401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:47.713517 401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:47.714250 401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:47.714466 401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:47.715162 401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:47.716774 401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:47.716831 401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.716872 401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:47.716887 401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.725785 401 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:42011
I20260808 06:03:47.725842 534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:42011 every 8 connection(s)
I20260808 06:03:47.726647 401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:47.735556 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.728482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:40371 (local address 127.13.229.126:36131)
0808 06:03:47.728614 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.728618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.729644 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 06:03:47.731075 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.731078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.731145 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.731237 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:47.732232 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.732768 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.733859 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.734067 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.734147 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.735145 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:47.735148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:47.735150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:47.735161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:47.735178 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:47.735185 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:47.735240 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.735386 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.735437 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:03:47.736045 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 401
I20260808 06:03:47.736181 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:47.736474 14229 external_mini_cluster.cc:1428] Setting key bde8178da280a0a2a3819e1c245f39e4
I20260808 06:03:47.736568 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.728883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36131 (local address 127.13.229.66:40371)
0808 06:03:47.729511 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.729551 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.730859 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.731392 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.731398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.731480 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:47.732074 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.732090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.732879 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.732881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.733753 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.733762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.734195 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.734198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.734353 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.734917 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:47.734918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:47.734957 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:47.735016 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:47.735259 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:47.735266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:47.735271 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.735362 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.735594 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260808 06:03:47.737761 535 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36131
I20260808 06:03:47.738008 535 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:47.738504 535 heartbeater.cc:511] Master 127.13.229.126:36131 requested a full tablet report, sending...
I20260808 06:03:47.738742 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--builtin_ntp_servers=127.13.229.84:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:47.739583 32670 ts_manager.cc:194] Registered new tserver with Master: 7e25633bfa22497092e48a6925115bfb (127.13.229.66:42011)
I20260808 06:03:47.740423 32670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.66:40371
W20260808 06:03:47.853526 539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:47.853838 539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:47.853905 539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:47.857882 539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:47.857957 539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:47.858047 539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:47.862819 539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:36131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:47.863926 539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:47.865471 539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:47.875447 544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.875504 547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:47.875511 545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:47.875696 539 server_base.cc:1034] running on GCE node
I20260808 06:03:47.876237 539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:47.876854 539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:47.878021 539 hybrid_clock.cc:648] HybridClock initialized: now 1786169027878006 us; error 25 us; skew 500 ppm
I20260808 06:03:47.879870 539 webserver.cc:460] Webserver started at http://127.13.229.67:38167/ using document root <none> and password file <none>
I20260808 06:03:47.880514 539 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:47.880575 539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:47.880831 539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:47.882858 539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "2f9dcaf832014a808018e7db9c88216b"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "22427cf3f37810f91bc6396ce8b6fa90"
server_key_iv: "4e979be905f66ffe305b42b04fa94cd5"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.883425 539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "2f9dcaf832014a808018e7db9c88216b"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "22427cf3f37810f91bc6396ce8b6fa90"
server_key_iv: "4e979be905f66ffe305b42b04fa94cd5"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.887064 539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:03:47.889647 553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.890831 539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:47.890956 539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "2f9dcaf832014a808018e7db9c88216b"
format_stamp: "Formatted at 2026-08-08 06:03:47 on dist-test-slave-zkpd"
server_key: "22427cf3f37810f91bc6396ce8b6fa90"
server_key_iv: "4e979be905f66ffe305b42b04fa94cd5"
server_key_version: "encryptionkey@0"
I20260808 06:03:47.891055 539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:47.902935 539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:47.903585 539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:47.903780 539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:47.904420 539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:47.905364 539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:47.905421 539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.905489 539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:47.905527 539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:47.914367 539 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:36027
I20260808 06:03:47.914390 671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:36027 every 8 connection(s)
I20260808 06:03:47.915310 539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:47.916440 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 539
I20260808 06:03:47.916584 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:47.916889 14229 external_mini_cluster.cc:1428] Setting key 086856d9d9523ad331ec1346c29cd0ba
I20260808 06:03:47.924803 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.917498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:36001 (local address 127.13.229.126:36131)
0808 06:03:47.917670 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.917675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.918937 (+ 1262us) server_negotiation.cc:407] Connection header received
0808 06:03:47.920430 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.920433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.920480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.920549 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:47.921372 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.921898 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.922991 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.923203 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.923278 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.924394 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:47.924397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:47.924399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:47.924420 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:47.924436 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:47.924444 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:47.924501 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.924641 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.924689 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 06:03:47.925652 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.918011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:36131 (local address 127.13.229.67:36001)
0808 06:03:47.918798 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.918852 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.920235 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.920679 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.920685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.920766 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:47.921258 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.921270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.922003 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.922005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.922887 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.922896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.923328 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.923331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.923477 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.924159 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:47.924161 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:47.924200 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:47.924261 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:47.924518 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:47.924525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:47.924530 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.924624 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.924835 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":152,"threads_started":1}
I20260808 06:03:47.926739 672 heartbeater.cc:347] Connected to a master server at 127.13.229.126:36131
I20260808 06:03:47.926983 672 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:47.927510 672 heartbeater.cc:511] Master 127.13.229.126:36131 requested a full tablet report, sending...
I20260808 06:03:47.928555 32670 ts_manager.cc:194] Registered new tserver with Master: 2f9dcaf832014a808018e7db9c88216b (127.13.229.67:36027)
I20260808 06:03:47.929091 32670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.67:36001
I20260808 06:03:47.930701 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:47.936122 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.932718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.13.229.126:36131)
0808 06:03:47.932834 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.932838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.933067 (+ 229us) server_negotiation.cc:407] Connection header received
0808 06:03:47.933152 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.933154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.933190 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.933254 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:47.933504 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.933991 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.934738 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.934936 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.935013 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.935730 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:47.935733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:47.935735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:47.935747 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:47.935762 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:47.935770 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:47.935827 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.935976 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.936022 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:03:47.938925 32670 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:47.942055 32670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:47.956458 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.950780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39321 (local address 127.0.0.1:58070)
0808 06:03:47.951288 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.951300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.951362 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.952351 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.952354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.952367 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:47.952485 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.952493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.953645 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.953650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.954870 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.954881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.955831 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.955834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.955988 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.956038 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.956166 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.956299 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":31,"thread_start_us":129,"threads_started":1}
I20260808 06:03:47.956493 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.951416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.13.229.65:39321)
0808 06:03:47.951848 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.951861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.951928 (+ 67us) server_negotiation.cc:407] Connection header received
0808 06:03:47.952010 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.952015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.952194 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.952376 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:47.952618 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.953540 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.955016 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.955712 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.955818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.956036 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.956208 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.956312 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":115,"threads_started":1}
I20260808 06:03:47.956638 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.950609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42011 (local address 127.0.0.1:46710)
0808 06:03:47.951176 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.951229 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.951362 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.952235 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.952243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.952271 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:47.952485 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.952493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.954123 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.954126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.955203 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.955212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.956278 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.956282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.956358 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.956380 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.956441 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.956511 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260808 06:03:47.957249 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.950786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.13.229.66:42011)
0808 06:03:47.951667 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.951683 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.951713 (+ 30us) server_negotiation.cc:407] Connection header received
0808 06:03:47.951797 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.951802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.952052 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.952670 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:47.952807 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.954000 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.955329 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.956148 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.956881 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.957023 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.957062 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.957128 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":145,"threads_started":1}
I20260808 06:03:47.958317 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.951273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:36027 (local address 127.0.0.1:50370)
0808 06:03:47.952366 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:47.952390 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:47.952568 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:47.953067 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:47.953070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:47.953089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:47.953173 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.953180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.954539 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.954542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.955572 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:47.955583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.957903 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.957907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:47.957991 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.958017 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 06:03:47.958087 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 06:03:47.958162 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":114,"threads_started":1}
I20260808 06:03:47.958894 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:47.951728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.13.229.67:36027)
0808 06:03:47.952211 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:47.952224 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:47.952467 (+ 243us) server_negotiation.cc:407] Connection header received
0808 06:03:47.952723 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:47.952728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:47.952916 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:47.953068 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:47.953297 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.954400 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.955711 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:47.956495 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:47.957643 (+ 1148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:47.957831 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:47.958557 (+ 726us) server_negotiation.cc:299] Negotiation successful
0808 06:03:47.958679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":188,"threads_started":1}
I20260808 06:03:47.959915 329 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 (DEFAULT_TABLE table=test-table [id=2e96737d64c04a26bc0265dc809b840d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:47.960296 467 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 (DEFAULT_TABLE table=test-table [id=2e96737d64c04a26bc0265dc809b840d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:47.961140 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce90c55fc2c403dadd1224ef6e916c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:47.961251 467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce90c55fc2c403dadd1224ef6e916c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:47.961416 604 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 (DEFAULT_TABLE table=test-table [id=2e96737d64c04a26bc0265dc809b840d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:47.962435 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce90c55fc2c403dadd1224ef6e916c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:47.968276 689 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Bootstrap starting.
I20260808 06:03:47.969254 690 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Bootstrap starting.
I20260808 06:03:47.970443 689 tablet_bootstrap.cc:654] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:47.971082 690 tablet_bootstrap.cc:654] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:47.971262 689 log.cc:824] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:47.971436 691 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Bootstrap starting.
I20260808 06:03:47.972075 690 log.cc:824] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:47.973513 689 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: No bootstrap required, opened a new log
I20260808 06:03:47.973697 691 tablet_bootstrap.cc:654] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:47.973714 689 ts_tablet_manager.cc:1397] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:47.973937 690 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: No bootstrap required, opened a new log
I20260808 06:03:47.974068 690 ts_tablet_manager.cc:1397] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 06:03:47.974543 691 log.cc:824] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:47.976487 691 tablet_bootstrap.cc:492] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: No bootstrap required, opened a new log
I20260808 06:03:47.976667 691 ts_tablet_manager.cc:1397] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:03:47.977550 690 raft_consensus.cc:348] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.977789 690 raft_consensus.cc:374] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:47.977841 690 raft_consensus.cc:729] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e25633bfa22497092e48a6925115bfb, State: Initialized, Role: FOLLOWER
I20260808 06:03:47.978039 689 raft_consensus.cc:348] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.978271 689 raft_consensus.cc:374] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:47.978320 689 raft_consensus.cc:729] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9dcaf832014a808018e7db9c88216b, State: Initialized, Role: FOLLOWER
I20260808 06:03:47.978308 690 consensus_queue.cc:260] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.978773 689 consensus_queue.cc:260] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.979384 690 ts_tablet_manager.cc:1428] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 06:03:47.979818 535 heartbeater.cc:503] Master 127.13.229.126:36131 was elected leader, sending a full tablet report...
I20260808 06:03:47.980890 689 ts_tablet_manager.cc:1428] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 06:03:47.980715 691 raft_consensus.cc:348] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.980973 691 raft_consensus.cc:374] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:47.981032 691 raft_consensus.cc:729] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14268b42bad4b3b9305b648154cb476, State: Initialized, Role: FOLLOWER
I20260808 06:03:47.981246 672 heartbeater.cc:503] Master 127.13.229.126:36131 was elected leader, sending a full tablet report...
I20260808 06:03:47.981449 691 consensus_queue.cc:260] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:47.982234 398 heartbeater.cc:503] Master 127.13.229.126:36131 was elected leader, sending a full tablet report...
I20260808 06:03:47.982296 691 ts_tablet_manager.cc:1428] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 06:03:48.019105 399 tablet.cc:2369] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:48.052165 696 raft_consensus.cc:482] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:48.052433 696 raft_consensus.cc:504] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:48.053637 696 leader_election.cc:288] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e25633bfa22497092e48a6925115bfb (127.13.229.66:42011), f14268b42bad4b3b9305b648154cb476 (127.13.229.65:39321)
I20260808 06:03:48.057272 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.054034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:56723 (local address 127.13.229.66:42011)
0808 06:03:48.054179 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:48.054186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:48.054205 (+ 19us) server_negotiation.cc:407] Connection header received
0808 06:03:48.054268 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:48.054271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:48.054333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:48.054439 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:48.054762 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.055268 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.056337 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.056715 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.056849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:48.056900 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:48.057083 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 06:03:48.057156 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:03:48.057276 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.053901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:42011 (local address 127.13.229.67:56723)
0808 06:03:48.054040 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:48.054056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:48.054150 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:48.054466 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:48.054469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:48.054513 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:48.054633 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:48.054640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.055390 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.055395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:48.056200 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:48.056208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.056842 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.056846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:48.056947 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:48.056975 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 06:03:48.057064 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 06:03:48.057135 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1,"spinlock_wait_cycles":4096}
I20260808 06:03:48.057518 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.054107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:56747 (local address 127.13.229.65:39321)
0808 06:03:48.054239 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:48.054245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:48.054464 (+ 219us) server_negotiation.cc:407] Connection header received
0808 06:03:48.054627 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:48.054630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:48.054686 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:48.054771 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:48.055027 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.055510 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.056585 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.056971 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.057104 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:48.057150 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:48.057340 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 06:03:48.057410 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 06:03:48.057544 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.053921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39321 (local address 127.13.229.67:56747)
0808 06:03:48.054367 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:48.054384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:48.054523 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:48.054783 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:48.054786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:48.054802 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:48.054917 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:48.054923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.055625 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.055628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:48.056461 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:48.056471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.057096 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.057099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:48.057206 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:48.057224 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 06:03:48.057307 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:03:48.057384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":110,"spinlock_wait_cycles":34560,"thread_start_us":135,"threads_started":1}
I20260808 06:03:48.058107 350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce90c55fc2c403dadd1224ef6e916c4" candidate_uuid: "2f9dcaf832014a808018e7db9c88216b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14268b42bad4b3b9305b648154cb476" is_pre_election: true
I20260808 06:03:48.058264 487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce90c55fc2c403dadd1224ef6e916c4" candidate_uuid: "2f9dcaf832014a808018e7db9c88216b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e25633bfa22497092e48a6925115bfb" is_pre_election: true
I20260808 06:03:48.058403 350 raft_consensus.cc:2454] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f9dcaf832014a808018e7db9c88216b in term 0.
I20260808 06:03:48.058588 487 raft_consensus.cc:2454] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f9dcaf832014a808018e7db9c88216b in term 0.
I20260808 06:03:48.058954 555 leader_election.cc:302] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f9dcaf832014a808018e7db9c88216b, f14268b42bad4b3b9305b648154cb476; no voters:
I20260808 06:03:48.059224 696 raft_consensus.cc:2781] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:48.059305 696 raft_consensus.cc:482] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:48.059336 696 raft_consensus.cc:3037] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:48.060426 696 raft_consensus.cc:504] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:48.060770 696 leader_election.cc:288] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [CANDIDATE]: Term 1 election: Requested vote from peers 7e25633bfa22497092e48a6925115bfb (127.13.229.66:42011), f14268b42bad4b3b9305b648154cb476 (127.13.229.65:39321)
I20260808 06:03:48.061141 487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce90c55fc2c403dadd1224ef6e916c4" candidate_uuid: "2f9dcaf832014a808018e7db9c88216b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e25633bfa22497092e48a6925115bfb"
I20260808 06:03:48.061141 350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce90c55fc2c403dadd1224ef6e916c4" candidate_uuid: "2f9dcaf832014a808018e7db9c88216b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14268b42bad4b3b9305b648154cb476"
I20260808 06:03:48.061268 350 raft_consensus.cc:3037] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:48.061268 487 raft_consensus.cc:3037] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:48.062541 487 raft_consensus.cc:2454] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f9dcaf832014a808018e7db9c88216b in term 1.
I20260808 06:03:48.062541 350 raft_consensus.cc:2454] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f9dcaf832014a808018e7db9c88216b in term 1.
I20260808 06:03:48.062873 556 leader_election.cc:302] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f9dcaf832014a808018e7db9c88216b, 7e25633bfa22497092e48a6925115bfb; no voters:
I20260808 06:03:48.063068 696 raft_consensus.cc:2781] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:48.063319 696 raft_consensus.cc:686] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 1 LEADER]: Becoming Leader. State: Replica: 2f9dcaf832014a808018e7db9c88216b, State: Running, Role: LEADER
I20260808 06:03:48.063664 696 consensus_queue.cc:237] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } }
I20260808 06:03:48.066774 32670 catalog_manager.cc:5495] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f9dcaf832014a808018e7db9c88216b (127.13.229.67). New cstate: current_term: 1 leader_uuid: "2f9dcaf832014a808018e7db9c88216b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f9dcaf832014a808018e7db9c88216b" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 36027 } health_report { overall_health: HEALTHY } } }
I20260808 06:03:48.077631 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.074034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.13.229.67:36027)
0808 06:03:48.074150 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:48.074155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:48.074198 (+ 43us) server_negotiation.cc:407] Connection header received
0808 06:03:48.074383 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:48.074385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:48.074443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:48.074551 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:48.074797 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.075279 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.076205 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:48.076426 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:48.076535 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:48.076845 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:48.077216 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:48.077354 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:48.077435 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 06:03:48.077482 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 06:03:48.083153 487 raft_consensus.cc:1261] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 1 FOLLOWER]: Refusing update from remote peer 2f9dcaf832014a808018e7db9c88216b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:48.083214 350 raft_consensus.cc:1261] T 9ce90c55fc2c403dadd1224ef6e916c4 P f14268b42bad4b3b9305b648154cb476 [term 1 FOLLOWER]: Refusing update from remote peer 2f9dcaf832014a808018e7db9c88216b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:48.083941 696 consensus_queue.cc:1035] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e25633bfa22497092e48a6925115bfb" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:48.084154 699 consensus_queue.cc:1035] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14268b42bad4b3b9305b648154cb476" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:48.091276 702 mvcc.cc:205] Tried to move back new op lower bound from 7316148339022282752 to 7316148338951561216. Current Snapshot: MvccSnapshot[applied={T|T < 7316148339022282752}]
I20260808 06:03:48.092342 704 mvcc.cc:205] Tried to move back new op lower bound from 7316148339022282752 to 7316148338951561216. Current Snapshot: MvccSnapshot[applied={T|T < 7316148339022282752}]
I20260808 06:03:48.101146 32669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:48.101286 32669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:48.103681 32669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a37ee68a01e46598cd0af14e90d6237: Sending DeleteTablet for 3 replicas of tablet 9ce90c55fc2c403dadd1224ef6e916c4
I20260808 06:03:48.104439 467 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:48 UTC) from {username='slave'} at 127.0.0.1:46710
I20260808 06:03:48.104661 329 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:48 UTC) from {username='slave'} at 127.0.0.1:58070
I20260808 06:03:48.104682 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce90c55fc2c403dadd1224ef6e916c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:48 UTC) from {username='slave'} at 127.0.0.1:50370
I20260808 06:03:48.105141 714 tablet_replica.cc:330] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: stopping tablet replica
I20260808 06:03:48.105449 714 raft_consensus.cc:2229] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:48.105520 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32734
I20260808 06:03:48.107678 714 raft_consensus.cc:2258] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:48.109328 714 ts_tablet_manager.cc:1905] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:48.110107 712 tablet_replica.cc:330] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: stopping tablet replica
I20260808 06:03:48.110430 712 raft_consensus.cc:2229] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:48.110759 712 raft_consensus.cc:2258] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:48.112401 712 ts_tablet_manager.cc:1905] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:48.112665 714 ts_tablet_manager.cc:1918] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:48.112775 714 log.cc:1196] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/9ce90c55fc2c403dadd1224ef6e916c4
I20260808 06:03:48.113111 714 ts_tablet_manager.cc:1939] T 9ce90c55fc2c403dadd1224ef6e916c4 P 2f9dcaf832014a808018e7db9c88216b: Deleting consensus metadata
I20260808 06:03:48.114434 32655 catalog_manager.cc:4841] TS 2f9dcaf832014a808018e7db9c88216b (127.13.229.67:36027): tablet 9ce90c55fc2c403dadd1224ef6e916c4 (table test-table [id=2e96737d64c04a26bc0265dc809b840d]) successfully deleted
I20260808 06:03:48.115352 712 ts_tablet_manager.cc:1918] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:48.115451 712 log.cc:1196] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/9ce90c55fc2c403dadd1224ef6e916c4
I20260808 06:03:48.115736 712 ts_tablet_manager.cc:1939] T 9ce90c55fc2c403dadd1224ef6e916c4 P 7e25633bfa22497092e48a6925115bfb: Deleting consensus metadata
I20260808 06:03:48.116945 32657 catalog_manager.cc:4841] TS 7e25633bfa22497092e48a6925115bfb (127.13.229.66:42011): tablet 9ce90c55fc2c403dadd1224ef6e916c4 (table test-table [id=2e96737d64c04a26bc0265dc809b840d]) successfully deleted
W20260808 06:03:48.117974 32656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:39321 (error 108)
I20260808 06:03:48.118860 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 401
I20260808 06:03:48.119045 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:48.118728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:39321 (local address 127.0.0.1:58072)
0808 06:03:48.118902 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:48.118969 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260808 06:03:48.124778 32656 catalog_manager.cc:4584] TS f14268b42bad4b3b9305b648154cb476 (127.13.229.65:39321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ce90c55fc2c403dadd1224ef6e916c4: Network error: Client connection negotiation failed: client connection to 127.13.229.65:39321: connect: Connection refused (error 111)
I20260808 06:03:48.131809 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 539
I20260808 06:03:48.141371 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 32640
2026-08-08T06:03:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:48 dist-test-slave-zkpd krb5kdc[718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:48 dist-test-slave-zkpd krb5kdc[718](info): set up 2 sockets
Aug 08 06:03:48 dist-test-slave-zkpd krb5kdc[718](info): commencing operation
krb5kdc: starting...
W20260808 06:03:49.205510 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:03:49.328987 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:35035
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:49.441540 731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:49.441780 731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:49.441838 731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:49.445878 731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:49.445927 731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:49.445946 731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:49.445963 731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:49.445977 731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:49.451040 731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:49.452076 731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:49.453074 731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:49.463572 739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:49.463572 736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.463970 731 server_base.cc:1034] running on GCE node
W20260808 06:03:49.464354 737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.465183 731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:49.466816 731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:49.468014 731 hybrid_clock.cc:648] HybridClock initialized: now 1786169029467963 us; error 70 us; skew 500 ppm
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:49.475255 731 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:03:49.476259 731 webserver.cc:460] Webserver started at http://127.13.229.126:44097/ using document root <none> and password file <none>
I20260808 06:03:49.476802 731 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:49.476861 731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:49.477029 731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:49.479066 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "ef5f45957d964a6f9835c9172c644601"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "c1383b4e370a09c863fd7b0aaab61424"
server_key_iv: "2b196915cfd44110586a71d1d112e56e"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.479578 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "ef5f45957d964a6f9835c9172c644601"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "c1383b4e370a09c863fd7b0aaab61424"
server_key_iv: "2b196915cfd44110586a71d1d112e56e"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.483304 731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 06:03:49.485679 746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.486901 731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:03:49.487047 731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "ef5f45957d964a6f9835c9172c644601"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "c1383b4e370a09c863fd7b0aaab61424"
server_key_iv: "2b196915cfd44110586a71d1d112e56e"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.487161 731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:49.505812 731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:49.508723 731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:49.508929 731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:49.516258 731 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:35035
I20260808 06:03:49.516280 799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:35035 every 8 connection(s)
I20260808 06:03:49.517181 731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:49.519847 800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:49.525409 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 731
I20260808 06:03:49.525416 800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: Bootstrap starting.
I20260808 06:03:49.525568 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:49.525827 14229 external_mini_cluster.cc:1428] Setting key eb1211641d2023e249d75120809c3e0e
I20260808 06:03:49.528194 800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:49.529104 800 log.cc:824] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:49.531360 800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: No bootstrap required, opened a new log
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:49.535460 800 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } }
I20260808 06:03:49.535724 800 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:49.535809 800 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5f45957d964a6f9835c9172c644601, State: Initialized, Role: FOLLOWER
I20260808 06:03:49.536290 800 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } }
I20260808 06:03:49.536465 800 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:49.536553 800 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:49.536670 800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:49.537979 800 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } }
I20260808 06:03:49.538378 800 leader_election.cc:302] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef5f45957d964a6f9835c9172c644601; no voters:
I20260808 06:03:49.538720 800 leader_election.cc:288] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:49.538839 805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:49.539088 805 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [term 1 LEADER]: Becoming Leader. State: Replica: ef5f45957d964a6f9835c9172c644601, State: Running, Role: LEADER
I20260808 06:03:49.539391 805 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } }
I20260808 06:03:49.540144 800 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:49.541052 806 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef5f45957d964a6f9835c9172c644601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } } }
I20260808 06:03:49.541195 806 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:49.541718 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef5f45957d964a6f9835c9172c644601. Latest consensus state: current_term: 1 leader_uuid: "ef5f45957d964a6f9835c9172c644601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef5f45957d964a6f9835c9172c644601" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 35035 } } }
I20260808 06:03:49.542507 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:49.542508 814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:49.542805 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:49.527192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.13.229.126:35035)
0808 06:03:49.527649 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:49.527667 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:49.527711 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:03:49.528674 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:49.528698 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:49.529136 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:49.529515 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:49.530384 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.531821 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.532701 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.533417 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.533573 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:49.536368 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:49.536374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:49.536391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:49.536428 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:49.538453 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:49.539031 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:49.539037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:49.539043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:49.539251 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:49.539632 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:49.539635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:49.539637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:49.540173 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:49.540469 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:49.541114 (+ 645us) server_negotiation.cc:299] Negotiation successful
0808 06:03:49.541427 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":135,"threads_started":1}
I20260808 06:03:49.545431 814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:49.550462 814 catalog_manager.cc:1347] Generated new cluster ID: 38751fab77234c259110ec875761d34d
I20260808 06:03:49.550523 814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:49.563099 814 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:49.564056 814 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:49.569656 814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: Generated new TSK 0
I20260808 06:03:49.570288 814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:03:49.601233 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:49.712574 826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:49.712839 826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:49.712917 826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:49.716821 826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:49.716905 826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:49.716995 826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:49.722010 826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:49.722983 826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:49.723979 826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:49.733202 831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:49.733258 832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.733335 826 server_base.cc:1034] running on GCE node
W20260808 06:03:49.733541 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.733965 826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:49.734562 826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:49.735719 826 hybrid_clock.cc:648] HybridClock initialized: now 1786169029735701 us; error 29 us; skew 500 ppm
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:49.738561 826 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:03:49.739393 826 webserver.cc:460] Webserver started at http://127.13.229.65:33003/ using document root <none> and password file <none>
I20260808 06:03:49.740017 826 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:49.740083 826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:49.740377 826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:49.742594 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "e2586f23e4174e29bf263da15d01d028"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "1dea9a4f981317ae4dca07e4d99b3a41"
server_key_iv: "3e706837809540472539082abb97105f"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.743171 826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "e2586f23e4174e29bf263da15d01d028"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "1dea9a4f981317ae4dca07e4d99b3a41"
server_key_iv: "3e706837809540472539082abb97105f"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.746869 826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:03:49.749243 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.750437 826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:03:49.750558 826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "e2586f23e4174e29bf263da15d01d028"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "1dea9a4f981317ae4dca07e4d99b3a41"
server_key_iv: "3e706837809540472539082abb97105f"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.750658 826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:49.762463 826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:49.765439 826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:49.765643 826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:49.766247 826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:49.767083 826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:49.767130 826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.767202 826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:49.767232 826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.776580 826 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:37959
I20260808 06:03:49.776600 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:37959 every 8 connection(s)
I20260808 06:03:49.777608 826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:49.777997 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 826
I20260808 06:03:49.778146 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:49.778509 14229 external_mini_cluster.cc:1428] Setting key 37c0b065b2393d8467e02dcef3b1106b
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
I20260808 06:03:49.793033 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:49.779695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:37409 (local address 127.13.229.126:35035)
0808 06:03:49.779836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:49.779839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:49.780980 (+ 1141us) server_negotiation.cc:407] Connection header received
0808 06:03:49.782409 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:49.782412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:49.782466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:49.782549 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:49.783972 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.784649 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.786411 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.786666 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.786786 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:49.789346 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:49.789351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:49.789356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:49.789395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:49.790840 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:49.791415 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:49.791418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:49.791419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:49.791498 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:49.791866 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:49.791868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:49.791869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:49.792111 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:49.792219 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:49.792789 (+ 570us) server_negotiation.cc:299] Negotiation successful
0808 06:03:49.792901 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:03:49.794286 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:49.780294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35035 (local address 127.13.229.65:37409)
0808 06:03:49.780794 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:49.780849 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:49.782178 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:49.782727 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:49.782734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:49.783104 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:49.783732 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:49.783757 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.784811 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.784818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:49.786274 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:49.786285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:49.786828 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:49.786832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:49.787007 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:49.787635 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:49.787653 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:49.789163 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:49.790949 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:49.790953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:49.790966 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:49.791312 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:49.791593 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:49.791599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:49.791601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:49.791781 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:49.792222 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:49.792227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:49.792541 (+ 314us) client_negotiation.cc:769] Sending connection context
0808 06:03:49.792732 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 06:03:49.793116 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260808 06:03:49.795626 958 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35035
I20260808 06:03:49.795933 958 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:49.796432 958 heartbeater.cc:511] Master 127.13.229.126:35035 requested a full tablet report, sending...
I20260808 06:03:49.797911 763 ts_manager.cc:194] Registered new tserver with Master: e2586f23e4174e29bf263da15d01d028 (127.13.229.65:37959)
I20260808 06:03:49.799108 763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:37409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:03:49.810598 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:49.923792 963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:49.924137 963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:49.924218 963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:49.928220 963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:49.928318 963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:49.928413 963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:49.933938 963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:49.935016 963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:49.936134 963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:49.946676 969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.946797 963 server_base.cc:1034] running on GCE node
W20260808 06:03:49.946656 968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:49.946657 971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:49.947345 963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:49.948112 963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:49.949314 963 hybrid_clock.cc:648] HybridClock initialized: now 1786169029949297 us; error 26 us; skew 500 ppm
Aug 08 06:03:49 dist-test-slave-zkpd krb5kdc[718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:49.952188 963 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:03:49.952993 963 webserver.cc:460] Webserver started at http://127.13.229.66:33773/ using document root <none> and password file <none>
I20260808 06:03:49.953526 963 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:49.953586 963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:49.953781 963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:49.955854 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "753bfe57d0ea47aebfe25a2206c8eecc"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "9f17c7e4df2bd394fd2a5e51b2c7e574"
server_key_iv: "d7e1ea69061ef769662b3c32ef6308cb"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.956462 963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "753bfe57d0ea47aebfe25a2206c8eecc"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "9f17c7e4df2bd394fd2a5e51b2c7e574"
server_key_iv: "d7e1ea69061ef769662b3c32ef6308cb"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.960299 963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260808 06:03:49.962677 978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.964040 963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 06:03:49.964140 963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "753bfe57d0ea47aebfe25a2206c8eecc"
format_stamp: "Formatted at 2026-08-08 06:03:49 on dist-test-slave-zkpd"
server_key: "9f17c7e4df2bd394fd2a5e51b2c7e574"
server_key_iv: "d7e1ea69061ef769662b3c32ef6308cb"
server_key_version: "encryptionkey@0"
I20260808 06:03:49.964228 963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:49.989732 963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:49.993391 963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:49.993625 963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:49.994212 963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:49.995034 963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:49.995086 963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:49.995127 963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:49.995148 963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.003813 963 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:43035
I20260808 06:03:50.003963 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:43035 every 8 connection(s)
I20260808 06:03:50.004781 963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:50.007570 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 963
I20260808 06:03:50.007710 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:50.008085 14229 external_mini_cluster.cc:1428] Setting key b53dedcef501f9bed700747b98edcf5e
Aug 08 06:03:50 dist-test-slave-zkpd krb5kdc[718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169029, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
I20260808 06:03:50.019673 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.006597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:59589 (local address 127.13.229.126:35035)
0808 06:03:50.006737 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.006741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.007536 (+ 795us) server_negotiation.cc:407] Connection header received
0808 06:03:50.009230 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.009233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.009287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.009369 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:50.010594 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.011121 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.012679 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.012918 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.013005 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.016028 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:50.016031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:50.016033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:50.016056 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:50.017583 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.018124 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.018127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.018129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.018210 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.018560 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.018563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.018565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.018744 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:50.018851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.019380 (+ 529us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.019501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:03:50.020562 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.006915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35035 (local address 127.13.229.66:59589)
0808 06:03:50.007378 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.007422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.009018 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.009550 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.009560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.009969 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:50.010444 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.010459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.011249 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.011252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.012550 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.012566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.013079 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.013086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.013320 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.013947 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:50.013973 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:50.015794 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:50.017720 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:50.017725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:50.017740 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:50.018013 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:50.018306 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:50.018309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:50.018311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:50.018455 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:50.018851 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:50.018859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:50.019133 (+ 274us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.019315 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.019627 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
I20260808 06:03:50.021734 1095 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35035
I20260808 06:03:50.021983 1095 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:50.022491 1095 heartbeater.cc:511] Master 127.13.229.126:35035 requested a full tablet report, sending...
I20260808 06:03:50.023628 763 ts_manager.cc:194] Registered new tserver with Master: 753bfe57d0ea47aebfe25a2206c8eecc (127.13.229.66:43035)
I20260808 06:03:50.024276 763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:59589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:03:50.037148 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:50.153378 1100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:50.153615 1100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:50.153657 1100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:50.157708 1100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:50.157761 1100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:50.157816 1100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:50.162850 1100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:35035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:50.163841 1100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:50.164921 1100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:50.174711 1105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:50.174731 1106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:50.174712 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:50.174940 1100 server_base.cc:1034] running on GCE node
I20260808 06:03:50.175338 1100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:50.176010 1100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:50.177175 1100 hybrid_clock.cc:648] HybridClock initialized: now 1786169030177153 us; error 32 us; skew 500 ppm
Aug 08 06:03:50 dist-test-slave-zkpd krb5kdc[718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169030, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:50.180016 1100 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:03:50.180848 1100 webserver.cc:460] Webserver started at http://127.13.229.67:34093/ using document root <none> and password file <none>
I20260808 06:03:50.181396 1100 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:50.181456 1100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:50.181690 1100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:50.183694 1100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "4cb25bd7dd1545129a04431b23ca035f"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "d0de772d2dbf8442fe2bf317cbb399e8"
server_key_iv: "de09f6f41cc4c932170f7772f6b75eec"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.184322 1100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "4cb25bd7dd1545129a04431b23ca035f"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "d0de772d2dbf8442fe2bf317cbb399e8"
server_key_iv: "de09f6f41cc4c932170f7772f6b75eec"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.188032 1100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:03:50.190348 1116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.191624 1100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:50.191742 1100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "4cb25bd7dd1545129a04431b23ca035f"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "d0de772d2dbf8442fe2bf317cbb399e8"
server_key_iv: "de09f6f41cc4c932170f7772f6b75eec"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.191839 1100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:50.202319 1100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:50.205503 1100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:50.205709 1100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:50.206290 1100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:50.207113 1100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:50.207159 1100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.207226 1100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:50.207262 1100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.216151 1100 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:41587
I20260808 06:03:50.216238 1230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:41587 every 8 connection(s)
I20260808 06:03:50.217097 1100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:50.224722 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1100
I20260808 06:03:50.224870 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:50.225178 14229 external_mini_cluster.cc:1428] Setting key faf45d070795ae68d401d93de199b3c2
Aug 08 06:03:50 dist-test-slave-zkpd krb5kdc[718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169030, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:50.231195 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.218977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:58221 (local address 127.13.229.126:35035)
0808 06:03:50.219129 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.219133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.220020 (+ 887us) server_negotiation.cc:407] Connection header received
0808 06:03:50.221405 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.221408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.221466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.221578 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:50.222754 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.223314 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.224597 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.224887 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.225006 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.228066 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:50.228070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:50.228072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:50.228098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:50.229320 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.229824 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.229827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.229828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.229903 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.230237 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.230240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.230241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.230427 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:50.230507 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.230975 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.231076 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 06:03:50.232090 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.219363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:35035 (local address 127.13.229.67:58221)
0808 06:03:50.219822 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.219866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.221187 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.221768 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.221774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.222124 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:50.222607 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.222621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.223453 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.223456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.224471 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.224482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.225065 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.225070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.225238 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.226000 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:50.226024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:50.227840 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:50.229427 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:50.229432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:50.229445 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:50.229702 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:50.229983 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:50.229985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:50.229987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:50.230120 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:50.230519 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:50.230524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:50.230763 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.230923 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.231181 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260808 06:03:50.233166 1231 heartbeater.cc:347] Connected to a master server at 127.13.229.126:35035
I20260808 06:03:50.233414 1231 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:50.233923 1231 heartbeater.cc:511] Master 127.13.229.126:35035 requested a full tablet report, sending...
I20260808 06:03:50.234908 763 ts_manager.cc:194] Registered new tserver with Master: 4cb25bd7dd1545129a04431b23ca035f (127.13.229.67:41587)
I20260808 06:03:50.235462 763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:58221
I20260808 06:03:50.238969 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:50.248198 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.241036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.13.229.126:35035)
0808 06:03:50.241152 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.241158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.241331 (+ 173us) server_negotiation.cc:407] Connection header received
0808 06:03:50.241454 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.241456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.241490 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.241584 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:50.242200 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.242744 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.243528 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.243726 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.243799 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.245244 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:50.245247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:50.245249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:50.245271 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:50.246671 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.247105 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.247107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.247108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.247175 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:50.247435 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:50.247437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:50.247438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:50.247609 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:50.247719 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.248013 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.248086 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 06:03:50.251011 763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:50.254010 763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:50.268361 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.262929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:41587 (local address 127.0.0.1:44326)
0808 06:03:50.263365 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.263387 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.263480 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.264279 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.264281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.264311 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:50.264441 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.264448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265582 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.265587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.266807 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.266817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.267794 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.267798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.267929 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.267974 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.268087 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.268206 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":13,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260808 06:03:50.268376 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.263285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.13.229.67:41587)
0808 06:03:50.263756 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.263768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.263791 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:03:50.263873 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.263878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.264126 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.264595 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:50.264740 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265453 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.266940 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.267655 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.267767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.267951 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.268134 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.268232 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":158,"threads_started":1}
I20260808 06:03:50.269070 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.262211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:43035 (local address 127.0.0.1:46388)
0808 06:03:50.262713 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.262759 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.262885 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.264246 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.264253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.264275 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:50.264413 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.264427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265924 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.265929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.267377 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.267384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.268636 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.268639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.268729 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.268752 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.268836 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.268919 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":130,"thread_start_us":73,"threads_started":1}
I20260808 06:03:50.269107 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.262365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.13.229.66:43035)
0808 06:03:50.263761 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.263771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.263791 (+ 20us) server_negotiation.cc:407] Connection header received
0808 06:03:50.263856 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.263860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.264086 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.264258 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:50.264538 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265746 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.267505 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.268395 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.268636 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.268779 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.268876 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.268972 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":142,"threads_started":1}
I20260808 06:03:50.269251 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.262417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:37959 (local address 127.0.0.1:50962)
0808 06:03:50.263241 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.263255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.263345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.264430 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.264432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.264448 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:50.264537 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.264543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265952 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.265954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.267358 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.267370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.268733 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.268736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.268955 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.268986 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.269055 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.269128 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":114,"thread_start_us":91,"threads_started":1}
I20260808 06:03:50.269721 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.262962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.13.229.65:37959)
0808 06:03:50.263805 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.263822 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.263842 (+ 20us) server_negotiation.cc:407] Connection header received
0808 06:03:50.263928 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.263933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.264102 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.264261 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:50.264642 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.265797 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.267515 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.268416 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.268538 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.268674 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.269468 (+ 794us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.269570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":78,"threads_started":1}
I20260808 06:03:50.271948 1165 tablet_service.cc:1467] Processing CreateTablet for tablet 3c129ae80dc94aac997485792afc5935 (DEFAULT_TABLE table=test-table [id=243b1d820b2d41c680f74d5ce3227bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:50.272538 892 tablet_service.cc:1467] Processing CreateTablet for tablet 3c129ae80dc94aac997485792afc5935 (DEFAULT_TABLE table=test-table [id=243b1d820b2d41c680f74d5ce3227bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:50.273169 1165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c129ae80dc94aac997485792afc5935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:50.273582 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c129ae80dc94aac997485792afc5935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:50.273572 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3c129ae80dc94aac997485792afc5935 (DEFAULT_TABLE table=test-table [id=243b1d820b2d41c680f74d5ce3227bdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:50.274453 1028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c129ae80dc94aac997485792afc5935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:50.280159 1248 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Bootstrap starting.
I20260808 06:03:50.281373 1249 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Bootstrap starting.
I20260808 06:03:50.281701 1250 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: Bootstrap starting.
I20260808 06:03:50.283267 1249 tablet_bootstrap.cc:654] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:50.283407 1248 tablet_bootstrap.cc:654] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:50.283665 1250 tablet_bootstrap.cc:654] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:50.284538 1249 log.cc:824] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:50.284824 1248 log.cc:824] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:50.284920 1250 log.cc:824] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:50.286790 1249 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: No bootstrap required, opened a new log
I20260808 06:03:50.286900 1248 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: No bootstrap required, opened a new log
I20260808 06:03:50.286900 1250 tablet_bootstrap.cc:492] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: No bootstrap required, opened a new log
I20260808 06:03:50.287047 1249 ts_tablet_manager.cc:1397] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:50.287096 1248 ts_tablet_manager.cc:1397] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 06:03:50.287096 1250 ts_tablet_manager.cc:1397] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:50.289973 1249 raft_consensus.cc:348] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.290218 1249 raft_consensus.cc:374] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:50.290273 1249 raft_consensus.cc:729] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb25bd7dd1545129a04431b23ca035f, State: Initialized, Role: FOLLOWER
I20260808 06:03:50.290835 1249 consensus_queue.cc:260] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.290984 1250 raft_consensus.cc:348] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.290984 1248 raft_consensus.cc:348] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.291246 1250 raft_consensus.cc:374] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:50.291246 1248 raft_consensus.cc:374] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:50.291307 1248 raft_consensus.cc:729] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753bfe57d0ea47aebfe25a2206c8eecc, State: Initialized, Role: FOLLOWER
I20260808 06:03:50.291307 1250 raft_consensus.cc:729] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2586f23e4174e29bf263da15d01d028, State: Initialized, Role: FOLLOWER
I20260808 06:03:50.291746 1248 consensus_queue.cc:260] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.291746 1250 consensus_queue.cc:260] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.292049 1249 ts_tablet_manager.cc:1428] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260808 06:03:50.292542 1231 heartbeater.cc:503] Master 127.13.229.126:35035 was elected leader, sending a full tablet report...
I20260808 06:03:50.292733 1250 ts_tablet_manager.cc:1428] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:03:50.292733 1248 ts_tablet_manager.cc:1428] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:03:50.293097 958 heartbeater.cc:503] Master 127.13.229.126:35035 was elected leader, sending a full tablet report...
I20260808 06:03:50.293251 1095 heartbeater.cc:503] Master 127.13.229.126:35035 was elected leader, sending a full tablet report...
I20260808 06:03:50.311265 1254 raft_consensus.cc:482] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:50.311453 1254 raft_consensus.cc:504] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.312597 1254 leader_election.cc:288] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 753bfe57d0ea47aebfe25a2206c8eecc (127.13.229.66:43035), e2586f23e4174e29bf263da15d01d028 (127.13.229.65:37959)
I20260808 06:03:50.317818 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.312782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:43035 (local address 127.13.229.67:36759)
0808 06:03:50.312976 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.312991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.313082 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.313383 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.313385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.313399 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:50.313565 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.313572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.314733 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.314736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.316368 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.316381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.317248 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.317252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.317450 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.317501 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.317579 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.317685 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":300}
I20260808 06:03:50.317835 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.312865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:36759 (local address 127.13.229.66:43035)
0808 06:03:50.313006 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.313010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.313028 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:03:50.313195 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.313198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.313241 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.313344 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:50.313680 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.314593 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.316517 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.317135 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.317246 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.317313 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.317661 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.317736 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:03:50.317835 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.312902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:38523 (local address 127.13.229.65:37959)
0808 06:03:50.313019 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.313025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.313285 (+ 260us) server_negotiation.cc:407] Connection header received
0808 06:03:50.313359 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.313361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.313409 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.313485 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:50.313824 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.314623 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.316517 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.317115 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.317250 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.317301 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.317622 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.317714 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:03:50.318344 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.312821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:37959 (local address 127.13.229.67:38523)
0808 06:03:50.313186 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.313200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.313279 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.313502 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.313505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.313518 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:50.313689 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.313698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.314733 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.314736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.316368 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.316381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.317248 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.317252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.317444 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.317506 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.317579 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.318196 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":234,"thread_start_us":127,"threads_started":1}
I20260808 06:03:50.318724 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c129ae80dc94aac997485792afc5935" candidate_uuid: "4cb25bd7dd1545129a04431b23ca035f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" is_pre_election: true
I20260808 06:03:50.319057 1049 raft_consensus.cc:2454] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb25bd7dd1545129a04431b23ca035f in term 0.
I20260808 06:03:50.319078 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c129ae80dc94aac997485792afc5935" candidate_uuid: "4cb25bd7dd1545129a04431b23ca035f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2586f23e4174e29bf263da15d01d028" is_pre_election: true
I20260808 06:03:50.319398 912 raft_consensus.cc:2454] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cb25bd7dd1545129a04431b23ca035f in term 0.
I20260808 06:03:50.319572 1117 leader_election.cc:302] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb25bd7dd1545129a04431b23ca035f, 753bfe57d0ea47aebfe25a2206c8eecc; no voters:
I20260808 06:03:50.319928 1254 raft_consensus.cc:2781] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:50.320043 1254 raft_consensus.cc:482] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:50.320109 1254 raft_consensus.cc:3037] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:50.321163 1254 raft_consensus.cc:504] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.321538 1254 leader_election.cc:288] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [CANDIDATE]: Term 1 election: Requested vote from peers 753bfe57d0ea47aebfe25a2206c8eecc (127.13.229.66:43035), e2586f23e4174e29bf263da15d01d028 (127.13.229.65:37959)
I20260808 06:03:50.321864 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c129ae80dc94aac997485792afc5935" candidate_uuid: "4cb25bd7dd1545129a04431b23ca035f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753bfe57d0ea47aebfe25a2206c8eecc"
I20260808 06:03:50.321934 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c129ae80dc94aac997485792afc5935" candidate_uuid: "4cb25bd7dd1545129a04431b23ca035f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2586f23e4174e29bf263da15d01d028"
I20260808 06:03:50.321983 1049 raft_consensus.cc:3037] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:50.322085 912 raft_consensus.cc:3037] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:50.323112 1049 raft_consensus.cc:2454] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb25bd7dd1545129a04431b23ca035f in term 1.
I20260808 06:03:50.323218 912 raft_consensus.cc:2454] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cb25bd7dd1545129a04431b23ca035f in term 1.
I20260808 06:03:50.323444 1117 leader_election.cc:302] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cb25bd7dd1545129a04431b23ca035f, 753bfe57d0ea47aebfe25a2206c8eecc; no voters:
I20260808 06:03:50.323685 1254 raft_consensus.cc:2781] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:50.323971 1254 raft_consensus.cc:686] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 1 LEADER]: Becoming Leader. State: Replica: 4cb25bd7dd1545129a04431b23ca035f, State: Running, Role: LEADER
I20260808 06:03:50.324297 1254 consensus_queue.cc:237] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } }
I20260808 06:03:50.327266 763 catalog_manager.cc:5495] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb25bd7dd1545129a04431b23ca035f (127.13.229.67). New cstate: current_term: 1 leader_uuid: "4cb25bd7dd1545129a04431b23ca035f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cb25bd7dd1545129a04431b23ca035f" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 41587 } health_report { overall_health: HEALTHY } } }
I20260808 06:03:50.338366 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.335112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.13.229.67:41587)
0808 06:03:50.335245 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.335248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.335261 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:03:50.335370 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.335372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.335420 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.335502 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:50.335722 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.336258 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.337059 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.337260 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.337338 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.337595 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:50.338010 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:50.338118 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.338209 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.338244 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 06:03:50.344623 1049 raft_consensus.cc:1261] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 1 FOLLOWER]: Refusing update from remote peer 4cb25bd7dd1545129a04431b23ca035f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:50.344623 912 raft_consensus.cc:1261] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028 [term 1 FOLLOWER]: Refusing update from remote peer 4cb25bd7dd1545129a04431b23ca035f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:50.345405 1259 consensus_queue.cc:1035] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [LEADER]: Connected to new peer: Peer: permanent_uuid: "753bfe57d0ea47aebfe25a2206c8eecc" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:50.345561 1254 consensus_queue.cc:1035] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2586f23e4174e29bf263da15d01d028" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:50.354382 1264 mvcc.cc:205] Tried to move back new op lower bound from 7316148348284014592 to 7316148348210880512. Current Snapshot: MvccSnapshot[applied={T|T < 7316148348284014592}]
I20260808 06:03:50.356423 1263 mvcc.cc:205] Tried to move back new op lower bound from 7316148348284014592 to 7316148348210880512. Current Snapshot: MvccSnapshot[applied={T|T < 7316148348284014592}]
I20260808 06:03:50.362687 761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:50.362869 761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:50.365384 761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef5f45957d964a6f9835c9172c644601: Sending DeleteTablet for 3 replicas of tablet 3c129ae80dc94aac997485792afc5935
I20260808 06:03:50.366318 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c129ae80dc94aac997485792afc5935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:50962
I20260808 06:03:50.366323 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c129ae80dc94aac997485792afc5935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:46388
I20260808 06:03:50.366437 1165 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c129ae80dc94aac997485792afc5935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:50 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:44326
I20260808 06:03:50.366947 1275 tablet_replica.cc:330] T 3c129ae80dc94aac997485792afc5935 P e2586f23e4174e29bf263da15d01d028: stopping tablet replica
I20260808 06:03:50.367408 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 826
I20260808 06:03:50.369282 1274 tablet_replica.cc:330] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: stopping tablet replica
I20260808 06:03:50.369688 1274 raft_consensus.cc:2229] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:50.370149 1274 raft_consensus.cc:2258] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:50.370553 1273 tablet_replica.cc:330] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: stopping tablet replica
I20260808 06:03:50.370882 1273 raft_consensus.cc:2229] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:50.371191 1273 raft_consensus.cc:2258] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:50.372123 1274 ts_tablet_manager.cc:1905] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:50.374994 1274 ts_tablet_manager.cc:1918] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:50.375095 1274 log.cc:1196] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/3c129ae80dc94aac997485792afc5935
I20260808 06:03:50.375427 1274 ts_tablet_manager.cc:1939] T 3c129ae80dc94aac997485792afc5935 P 4cb25bd7dd1545129a04431b23ca035f: Deleting consensus metadata
I20260808 06:03:50.376931 747 catalog_manager.cc:4841] TS 4cb25bd7dd1545129a04431b23ca035f (127.13.229.67:41587): tablet 3c129ae80dc94aac997485792afc5935 (table test-table [id=243b1d820b2d41c680f74d5ce3227bdf]) successfully deleted
I20260808 06:03:50.377496 1273 ts_tablet_manager.cc:1905] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 06:03:50.379524 749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:37959 (error 108)
I20260808 06:03:50.380429 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 963
I20260808 06:03:50.380848 1273 ts_tablet_manager.cc:1918] T 3c129ae80dc94aac997485792afc5935 P 753bfe57d0ea47aebfe25a2206c8eecc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:50.381237 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.380323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:37959 (local address 127.0.0.1:41884)
0808 06:03:50.381083 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.381152 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:37959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669}
W20260808 06:03:50.383386 749 catalog_manager.cc:4584] TS e2586f23e4174e29bf263da15d01d028 (127.13.229.65:37959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c129ae80dc94aac997485792afc5935: Network error: Client connection negotiation failed: client connection to 127.13.229.65:37959: connect: Connection refused (error 111)
I20260808 06:03:50.388232 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1100
I20260808 06:03:50.389475 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.388399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:43035 (local address 127.0.0.1:47270)
0808 06:03:50.389314 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.389378 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.66:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814}
I20260808 06:03:50.395327 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 731
2026-08-08T06:03:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2255 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T06:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:50Z Disabled control of system clock
I20260808 06:03:50.428884 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37633
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:50.540580 1281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:50.540830 1281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:50.540918 1281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:50.544925 1281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:50.544981 1281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:50.544996 1281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:50.545048 1281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:50.545077 1281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:50.549726 1281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:37633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:50.550789 1281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:50.552196 1281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:50.562366 1287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:50.562372 1286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:50.562536 1289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:50.563393 1281 server_base.cc:1034] running on GCE node
I20260808 06:03:50.564109 1281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:50.565547 1281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:50.566751 1281 hybrid_clock.cc:648] HybridClock initialized: now 1786169030566739 us; error 29 us; skew 500 ppm
I20260808 06:03:50.568708 1281 webserver.cc:460] Webserver started at http://127.13.229.126:37255/ using document root <none> and password file <none>
I20260808 06:03:50.569304 1281 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:50.569402 1281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:50.569655 1281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:50.571722 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "516a280d3e2b47d9b294fc2c28ebf5fc"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "1ecc0d328c6e389908f4188ff6ac4335"
server_key_iv: "afd71d5ef91ff40d5c3c770f4bcb468b"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.572358 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "516a280d3e2b47d9b294fc2c28ebf5fc"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "1ecc0d328c6e389908f4188ff6ac4335"
server_key_iv: "afd71d5ef91ff40d5c3c770f4bcb468b"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.576143 1281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 06:03:50.578563 1295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.579840 1281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:50.580022 1281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "516a280d3e2b47d9b294fc2c28ebf5fc"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "1ecc0d328c6e389908f4188ff6ac4335"
server_key_iv: "afd71d5ef91ff40d5c3c770f4bcb468b"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.580150 1281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:50.610981 1281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:50.611768 1281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:50.612027 1281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:50.619299 1281 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:37633
I20260808 06:03:50.619307 1348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:37633 every 8 connection(s)
I20260808 06:03:50.620362 1281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:50.623596 1349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:50.625366 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1281
I20260808 06:03:50.625524 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:50.625808 14229 external_mini_cluster.cc:1428] Setting key 34e62718a64412b322de32a5dc86691f
I20260808 06:03:50.630739 1349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: Bootstrap starting.
I20260808 06:03:50.633420 1349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:50.634398 1349 log.cc:824] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:50.636412 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.627298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.13.229.126:37633)
0808 06:03:50.628023 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.628044 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.628090 (+ 46us) server_negotiation.cc:407] Connection header received
0808 06:03:50.628890 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.628916 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.629429 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.629813 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:50.630265 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.631662 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.632596 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.633277 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.633458 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.634301 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:50.634308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:50.634324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:50.634841 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:50.634904 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:50.634926 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:50.635004 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.635211 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.635477 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":118,"threads_started":1}
I20260808 06:03:50.636683 1349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: No bootstrap required, opened a new log
I20260808 06:03:50.640504 1349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } }
I20260808 06:03:50.640714 1349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:50.640748 1349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516a280d3e2b47d9b294fc2c28ebf5fc, State: Initialized, Role: FOLLOWER
I20260808 06:03:50.641247 1349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } }
I20260808 06:03:50.641444 1349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:50.641521 1349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:50.641630 1349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:50.642472 1349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } }
I20260808 06:03:50.642764 1349 leader_election.cc:302] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 516a280d3e2b47d9b294fc2c28ebf5fc; no voters:
I20260808 06:03:50.643009 1349 leader_election.cc:288] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:50.643133 1354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:50.643380 1354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [term 1 LEADER]: Becoming Leader. State: Replica: 516a280d3e2b47d9b294fc2c28ebf5fc, State: Running, Role: LEADER
I20260808 06:03:50.643685 1354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } }
I20260808 06:03:50.644289 1349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:50.645382 1356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 516a280d3e2b47d9b294fc2c28ebf5fc. Latest consensus state: current_term: 1 leader_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } } }
I20260808 06:03:50.645524 1356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:50.645854 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "516a280d3e2b47d9b294fc2c28ebf5fc" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37633 } } }
I20260808 06:03:50.645959 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:50.646059 1363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:50.649010 1363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:50.654451 1363 catalog_manager.cc:1347] Generated new cluster ID: 15384b310280479594315bf1c6f72020
I20260808 06:03:50.654524 1363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:50.666992 1363 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:50.668254 1363 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:50.677189 1363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: Generated new TSK 0
I20260808 06:03:50.677910 1363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:50.687783 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--builtin_ntp_servers=127.13.229.84:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:50.804754 1374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:50.805030 1374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:50.805099 1374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:50.809147 1374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:50.809206 1374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:50.809260 1374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:50.813941 1374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:50.815058 1374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:50.816566 1374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:50.826332 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:50.826524 1374 server_base.cc:1034] running on GCE node
W20260808 06:03:50.826534 1379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:50.826584 1382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:50.827018 1374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:50.827698 1374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:50.828852 1374 hybrid_clock.cc:648] HybridClock initialized: now 1786169030828847 us; error 72 us; skew 500 ppm
I20260808 06:03:50.830601 1374 webserver.cc:460] Webserver started at http://127.13.229.65:37073/ using document root <none> and password file <none>
I20260808 06:03:50.831162 1374 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:50.831223 1374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:50.831455 1374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:50.833562 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "3c14f988007b4d6b8c27ea8a47294235"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "876ddce219a3f7d27c9bc2d082ae042d"
server_key_iv: "9f01d45af75f3355d2778cf2d64483a4"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.834143 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "3c14f988007b4d6b8c27ea8a47294235"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "876ddce219a3f7d27c9bc2d082ae042d"
server_key_iv: "9f01d45af75f3355d2778cf2d64483a4"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.837770 1374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:03:50.840219 1389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.841398 1374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:03:50.841532 1374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "3c14f988007b4d6b8c27ea8a47294235"
format_stamp: "Formatted at 2026-08-08 06:03:50 on dist-test-slave-zkpd"
server_key: "876ddce219a3f7d27c9bc2d082ae042d"
server_key_iv: "9f01d45af75f3355d2778cf2d64483a4"
server_key_version: "encryptionkey@0"
I20260808 06:03:50.841638 1374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:50.861159 1374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:50.861846 1374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:50.862049 1374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:50.862648 1374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:50.863616 1374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:50.863685 1374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.863759 1374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:50.863801 1374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:50.872643 1374 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:41767
I20260808 06:03:50.872692 1505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:41767 every 8 connection(s)
I20260808 06:03:50.873553 1374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:50.875449 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1374
I20260808 06:03:50.875607 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:50.875980 14229 external_mini_cluster.cc:1428] Setting key ad47f6c83389ddf856b1e8faa8842e07
I20260808 06:03:50.878386 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--builtin_ntp_servers=127.13.229.84:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:50.883991 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.875448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:60117 (local address 127.13.229.126:37633)
0808 06:03:50.875635 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:50.875641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:50.876929 (+ 1288us) server_negotiation.cc:407] Connection header received
0808 06:03:50.878724 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:50.878727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:50.878772 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:50.878844 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:50.880091 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.880633 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.881822 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.882057 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.882170 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.883448 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:50.883451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:50.883453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:50.883466 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:50.883486 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:50.883494 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:50.883569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:50.883743 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 06:03:50.883816 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 06:03:50.884928 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:50.875914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37633 (local address 127.13.229.65:60117)
0808 06:03:50.876754 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:50.876812 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:50.878488 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:50.879039 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:50.879052 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:50.879175 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:50.879941 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.879967 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.880777 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.880781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.881684 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:50.881694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:50.882217 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:50.882221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:50.882404 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:50.883121 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:50.883122 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:50.883192 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:50.883285 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:50.883585 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:50.883594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:50.883600 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 06:03:50.883731 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 06:03:50.884070 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":132,"threads_started":1}
I20260808 06:03:50.886183 1506 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37633
I20260808 06:03:50.886442 1506 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:50.886977 1506 heartbeater.cc:511] Master 127.13.229.126:37633 requested a full tablet report, sending...
I20260808 06:03:50.888847 1309 ts_manager.cc:194] Registered new tserver with Master: 3c14f988007b4d6b8c27ea8a47294235 (127.13.229.65:41767)
I20260808 06:03:50.890659 1309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.65:60117
W20260808 06:03:50.994755 1509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:50.995002 1509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:50.995039 1509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:50.999042 1509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:50.999090 1509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:50.999141 1509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:51.003611 1509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:51.004645 1509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:51.006021 1509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:51.015441 1517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:51.015548 1515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:51.015688 1514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:51.015781 1509 server_base.cc:1034] running on GCE node
I20260808 06:03:51.016199 1509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:51.016832 1509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:51.018033 1509 hybrid_clock.cc:648] HybridClock initialized: now 1786169031018018 us; error 25 us; skew 500 ppm
I20260808 06:03:51.019968 1509 webserver.cc:460] Webserver started at http://127.13.229.66:41097/ using document root <none> and password file <none>
I20260808 06:03:51.020542 1509 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:51.020638 1509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:51.020874 1509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:51.022958 1509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "f022a886e7b1435eb8f7cb9cedb5996d"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "63835424c2bff030889779e374e202df"
server_key_iv: "d2b2079ee9cbadeb7c7be3bbccb11232"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.023547 1509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "f022a886e7b1435eb8f7cb9cedb5996d"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "63835424c2bff030889779e374e202df"
server_key_iv: "d2b2079ee9cbadeb7c7be3bbccb11232"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.027347 1509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 06:03:51.029907 1523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.031147 1509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:51.031283 1509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "f022a886e7b1435eb8f7cb9cedb5996d"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "63835424c2bff030889779e374e202df"
server_key_iv: "d2b2079ee9cbadeb7c7be3bbccb11232"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.031404 1509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:51.049671 1509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:51.050371 1509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:51.050577 1509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:51.051203 1509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:51.052234 1509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:51.052309 1509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.052384 1509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:51.052425 1509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.061367 1509 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35599
I20260808 06:03:51.061424 1639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35599 every 8 connection(s)
I20260808 06:03:51.062305 1509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:51.065864 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1509
I20260808 06:03:51.066021 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:51.066340 14229 external_mini_cluster.cc:1428] Setting key 49a97e0ee895da1aa2bd53c95ec828f5
I20260808 06:03:51.068599 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--builtin_ntp_servers=127.13.229.84:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:51.073007 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.064252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:47183 (local address 127.13.229.126:37633)
0808 06:03:51.064397 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.064401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.065473 (+ 1072us) server_negotiation.cc:407] Connection header received
0808 06:03:51.067430 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.067434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.067489 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.067596 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:51.068821 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.069644 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.070782 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.071172 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.071286 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.072409 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:51.072416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:51.072422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:51.072438 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:51.072462 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:51.072474 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:51.072553 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.072695 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.072763 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 06:03:51.073756 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.064669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37633 (local address 127.13.229.66:47183)
0808 06:03:51.065314 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.065356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.067198 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.067775 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.067782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.067890 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:51.068609 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.068636 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.069777 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.069780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.070657 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.070667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.071328 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.071331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.071478 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.072169 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:51.072172 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:51.072212 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:51.072271 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:51.072561 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:51.072569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:51.072573 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.072665 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.072880 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260808 06:03:51.074965 1640 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37633
I20260808 06:03:51.075193 1640 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:51.075711 1640 heartbeater.cc:511] Master 127.13.229.126:37633 requested a full tablet report, sending...
I20260808 06:03:51.076845 1309 ts_manager.cc:194] Registered new tserver with Master: f022a886e7b1435eb8f7cb9cedb5996d (127.13.229.66:35599)
I20260808 06:03:51.077419 1309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.66:47183
W20260808 06:03:51.183934 1643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:51.184207 1643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:51.184283 1643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:51.188275 1643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:51.188323 1643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:51.188413 1643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:51.193077 1643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:37633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:51.194104 1643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:51.195574 1643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:03:51.205098 1643 server_base.cc:1034] running on GCE node
W20260808 06:03:51.205063 1650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:51.205116 1649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:51.205241 1652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:51.205790 1643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:51.206399 1643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:51.207595 1643 hybrid_clock.cc:648] HybridClock initialized: now 1786169031207585 us; error 28 us; skew 500 ppm
I20260808 06:03:51.209496 1643 webserver.cc:460] Webserver started at http://127.13.229.67:44923/ using document root <none> and password file <none>
I20260808 06:03:51.210069 1643 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:51.210167 1643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:51.210409 1643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:51.212518 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "00e789b1e6c6404e8ccca611cc8f35f6"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "e1781794f30ca46953f40ece499b397b"
server_key_iv: "8798a9360ae7196a0bed18c69e2d7ec7"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.213083 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "00e789b1e6c6404e8ccca611cc8f35f6"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "e1781794f30ca46953f40ece499b397b"
server_key_iv: "8798a9360ae7196a0bed18c69e2d7ec7"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.216701 1643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:03:51.219023 1658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.220233 1643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:51.220368 1643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "00e789b1e6c6404e8ccca611cc8f35f6"
format_stamp: "Formatted at 2026-08-08 06:03:51 on dist-test-slave-zkpd"
server_key: "e1781794f30ca46953f40ece499b397b"
server_key_iv: "8798a9360ae7196a0bed18c69e2d7ec7"
server_key_version: "encryptionkey@0"
I20260808 06:03:51.220485 1643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:51.242587 1643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:51.243287 1643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:51.243496 1643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:51.244191 1643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:51.245261 1643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:51.245340 1643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.245433 1643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:51.245478 1643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:51.254256 1643 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:40585
I20260808 06:03:51.254334 1775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:40585 every 8 connection(s)
I20260808 06:03:51.255178 1643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:51.256325 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1643
I20260808 06:03:51.256443 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:51.256768 14229 external_mini_cluster.cc:1428] Setting key cb523dbed9268e4379de24e463b11351
I20260808 06:03:51.264739 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.257308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:45303 (local address 127.13.229.126:37633)
0808 06:03:51.257448 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.257452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.258824 (+ 1372us) server_negotiation.cc:407] Connection header received
0808 06:03:51.260401 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.260403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.260450 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.260518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:51.261350 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.261864 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.262958 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.263180 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.263258 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.264307 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:51.264310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:51.264313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:51.264324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:51.264343 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:51.264351 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:51.264440 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.264572 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.264625 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 06:03:51.265583 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.257841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:37633 (local address 127.13.229.67:45303)
0808 06:03:51.258677 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.258729 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.260210 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.260650 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.260656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.260738 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:51.261233 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.261246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.261988 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.261990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.262849 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.262858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.263311 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.263314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.263456 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.264062 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:51.264064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:51.264105 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:51.264169 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:51.264445 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:51.264453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:51.264457 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.264552 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.264775 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":162,"threads_started":1}
I20260808 06:03:51.266676 1776 heartbeater.cc:347] Connected to a master server at 127.13.229.126:37633
I20260808 06:03:51.266886 1776 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:51.267372 1776 heartbeater.cc:511] Master 127.13.229.126:37633 requested a full tablet report, sending...
I20260808 06:03:51.268445 1309 ts_manager.cc:194] Registered new tserver with Master: 00e789b1e6c6404e8ccca611cc8f35f6 (127.13.229.67:40585)
I20260808 06:03:51.269097 1309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.67:45303
I20260808 06:03:51.270974 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:51.276913 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.273153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.13.229.126:37633)
0808 06:03:51.273298 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.273301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.273477 (+ 176us) server_negotiation.cc:407] Connection header received
0808 06:03:51.273571 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.273574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.273615 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.273688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:51.273921 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.274496 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.275323 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.275553 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.275639 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.276479 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:51.276482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:51.276485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:51.276497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:51.276515 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:51.276524 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:51.276632 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.276741 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.276804 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:03:51.279829 1309 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:51.282992 1309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:51.298687 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.292166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.13.229.65:41767)
0808 06:03:51.293480 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.293493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.293517 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:03:51.293596 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.293604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.293800 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.293973 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:51.294208 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295308 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.296997 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.297840 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.297969 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.298174 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.298400 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.298511 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":107,"threads_started":1}
I20260808 06:03:51.298683 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.292238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:40585 (local address 127.0.0.1:47306)
0808 06:03:51.292724 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.292765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.292894 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.294135 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.294139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.294157 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:51.294230 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.294237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295319 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.295322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.296990 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.297001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.297982 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.297986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.298174 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.298240 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.298353 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.298508 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":259,"thread_start_us":81,"threads_started":1}
I20260808 06:03:51.298683 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.292473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41767 (local address 127.0.0.1:51606)
0808 06:03:51.293141 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.293151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.293217 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.293949 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.293952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.293970 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:51.294062 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.294070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295463 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.295466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.296853 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.296863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.297983 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.297986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.298178 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.298236 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.298353 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.298508 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":306,"thread_start_us":104,"threads_started":1}
I20260808 06:03:51.299404 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.292574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.13.229.67:40585)
0808 06:03:51.293535 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.293549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.293574 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:03:51.293652 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.293657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.293872 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.294021 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:51.294372 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295107 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.297126 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.297861 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.297969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.298173 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.299112 (+ 939us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.299217 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":136,"threads_started":1}
I20260808 06:03:51.300563 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.291876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35599 (local address 127.0.0.1:55590)
0808 06:03:51.292713 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.292765 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.292899 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.293648 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.293655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.293679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:51.293848 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.293855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295388 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.295391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.296990 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.297001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.300166 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.300169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.300266 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.300291 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.300353 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.300440 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":78,"thread_start_us":92,"threads_started":1}
I20260808 06:03:51.300916 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.292045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.13.229.66:35599)
0808 06:03:51.293151 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.293164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.293189 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:03:51.293274 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.293283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.293497 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.293662 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:51.294010 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.295223 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.298238 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.300025 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.300134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.300319 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.300374 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.300474 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":108,"threads_started":1}
I20260808 06:03:51.302155 1706 tablet_service.cc:1467] Processing CreateTablet for tablet d783751e1e2f4050896dd4e579fbfe58 (DEFAULT_TABLE table=test-table [id=492cff6c21d5418e923718fc46f75589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:51.301949 1438 tablet_service.cc:1467] Processing CreateTablet for tablet d783751e1e2f4050896dd4e579fbfe58 (DEFAULT_TABLE table=test-table [id=492cff6c21d5418e923718fc46f75589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:51.303217 1438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d783751e1e2f4050896dd4e579fbfe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:51.303326 1706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d783751e1e2f4050896dd4e579fbfe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:51.303555 1572 tablet_service.cc:1467] Processing CreateTablet for tablet d783751e1e2f4050896dd4e579fbfe58 (DEFAULT_TABLE table=test-table [id=492cff6c21d5418e923718fc46f75589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:51.304550 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d783751e1e2f4050896dd4e579fbfe58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:51.309621 1793 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Bootstrap starting.
I20260808 06:03:51.311086 1794 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Bootstrap starting.
I20260808 06:03:51.312166 1793 tablet_bootstrap.cc:654] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:51.312217 1795 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Bootstrap starting.
I20260808 06:03:51.313118 1793 log.cc:824] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:51.313469 1794 tablet_bootstrap.cc:654] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:51.314270 1795 tablet_bootstrap.cc:654] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:51.314314 1794 log.cc:824] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:51.315066 1795 log.cc:824] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:51.315455 1793 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: No bootstrap required, opened a new log
I20260808 06:03:51.315653 1793 ts_tablet_manager.cc:1397] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:51.316293 1794 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: No bootstrap required, opened a new log
I20260808 06:03:51.316505 1794 ts_tablet_manager.cc:1397] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260808 06:03:51.317104 1795 tablet_bootstrap.cc:492] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: No bootstrap required, opened a new log
I20260808 06:03:51.317281 1795 ts_tablet_manager.cc:1397] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 06:03:51.320115 1794 raft_consensus.cc:348] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.320374 1794 raft_consensus.cc:374] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:51.320431 1794 raft_consensus.cc:729] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c14f988007b4d6b8c27ea8a47294235, State: Initialized, Role: FOLLOWER
I20260808 06:03:51.320261 1793 raft_consensus.cc:348] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.320513 1793 raft_consensus.cc:374] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:51.320564 1793 raft_consensus.cc:729] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e789b1e6c6404e8ccca611cc8f35f6, State: Initialized, Role: FOLLOWER
I20260808 06:03:51.320901 1794 consensus_queue.cc:260] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.321051 1793 consensus_queue.cc:260] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.321411 1795 raft_consensus.cc:348] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.321645 1795 raft_consensus.cc:374] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:51.321709 1795 raft_consensus.cc:729] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f022a886e7b1435eb8f7cb9cedb5996d, State: Initialized, Role: FOLLOWER
I20260808 06:03:51.321846 1776 heartbeater.cc:503] Master 127.13.229.126:37633 was elected leader, sending a full tablet report...
I20260808 06:03:51.321985 1794 ts_tablet_manager.cc:1428] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 06:03:51.322042 1793 ts_tablet_manager.cc:1428] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:03:51.322129 1795 consensus_queue.cc:260] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.322316 1506 heartbeater.cc:503] Master 127.13.229.126:37633 was elected leader, sending a full tablet report...
I20260808 06:03:51.323067 1795 ts_tablet_manager.cc:1428] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260808 06:03:51.323403 1640 heartbeater.cc:503] Master 127.13.229.126:37633 was elected leader, sending a full tablet report...
W20260808 06:03:51.374815 1507 tablet.cc:2369] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:51.416764 1801 raft_consensus.cc:482] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:51.417037 1801 raft_consensus.cc:504] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.418558 1801 leader_election.cc:288] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c14f988007b4d6b8c27ea8a47294235 (127.13.229.65:41767), 00e789b1e6c6404e8ccca611cc8f35f6 (127.13.229.67:40585)
I20260808 06:03:51.423987 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.419000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:44629 (local address 127.13.229.67:40585)
0808 06:03:51.419131 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.419136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.419498 (+ 362us) server_negotiation.cc:407] Connection header received
0808 06:03:51.419610 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.419612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.419656 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.419718 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:51.420062 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.420778 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.422507 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.423169 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.423312 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.423366 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.423734 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.423822 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 06:03:51.423996 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.419102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:52695 (local address 127.13.229.65:41767)
0808 06:03:51.419255 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.419258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.419270 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:03:51.419319 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.419322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.419365 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.419431 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:51.419814 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.420670 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.422507 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.423175 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.423361 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.423420 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.423734 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.423863 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 06:03:51.423998 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.418791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:40585 (local address 127.13.229.66:44629)
0808 06:03:51.419369 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.419380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.419503 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.419765 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.419768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.419789 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:51.419884 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.419933 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.420882 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.420884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.422333 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.422346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.423439 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.423444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.423555 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.423611 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.423691 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.423815 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":192,"spinlock_wait_cycles":896,"thread_start_us":153,"threads_started":1}
I20260808 06:03:51.423969 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.418789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41767 (local address 127.13.229.66:52695)
0808 06:03:51.418957 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.418975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:51.419072 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:51.419476 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:51.419479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:51.419505 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:51.419661 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.419667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.420807 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.420810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.422333 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:51.422346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.423469 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.423472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:51.423558 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.423614 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 06:03:51.423691 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:03:51.423791 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":135,"spinlock_wait_cycles":4224}
I20260808 06:03:51.425072 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d783751e1e2f4050896dd4e579fbfe58" candidate_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" is_pre_election: true
I20260808 06:03:51.425108 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d783751e1e2f4050896dd4e579fbfe58" candidate_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c14f988007b4d6b8c27ea8a47294235" is_pre_election: true
I20260808 06:03:51.425401 1727 raft_consensus.cc:2454] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022a886e7b1435eb8f7cb9cedb5996d in term 0.
I20260808 06:03:51.425415 1459 raft_consensus.cc:2454] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f022a886e7b1435eb8f7cb9cedb5996d in term 0.
I20260808 06:03:51.425918 1527 leader_election.cc:302] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e789b1e6c6404e8ccca611cc8f35f6, 3c14f988007b4d6b8c27ea8a47294235, f022a886e7b1435eb8f7cb9cedb5996d; no voters:
I20260808 06:03:51.426205 1801 raft_consensus.cc:2781] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:51.426288 1801 raft_consensus.cc:482] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:51.426339 1801 raft_consensus.cc:3037] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:51.427445 1801 raft_consensus.cc:504] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.427814 1801 leader_election.cc:288] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [CANDIDATE]: Term 1 election: Requested vote from peers 3c14f988007b4d6b8c27ea8a47294235 (127.13.229.65:41767), 00e789b1e6c6404e8ccca611cc8f35f6 (127.13.229.67:40585)
I20260808 06:03:51.428234 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d783751e1e2f4050896dd4e579fbfe58" candidate_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c14f988007b4d6b8c27ea8a47294235"
I20260808 06:03:51.428356 1727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d783751e1e2f4050896dd4e579fbfe58" candidate_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e789b1e6c6404e8ccca611cc8f35f6"
I20260808 06:03:51.428392 1459 raft_consensus.cc:3037] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:51.428486 1727 raft_consensus.cc:3037] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:51.429752 1459 raft_consensus.cc:2454] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022a886e7b1435eb8f7cb9cedb5996d in term 1.
I20260808 06:03:51.429752 1727 raft_consensus.cc:2454] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f022a886e7b1435eb8f7cb9cedb5996d in term 1.
I20260808 06:03:51.430138 1526 leader_election.cc:302] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c14f988007b4d6b8c27ea8a47294235, f022a886e7b1435eb8f7cb9cedb5996d; no voters:
I20260808 06:03:51.430348 1801 raft_consensus.cc:2781] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:51.430598 1801 raft_consensus.cc:686] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 1 LEADER]: Becoming Leader. State: Replica: f022a886e7b1435eb8f7cb9cedb5996d, State: Running, Role: LEADER
I20260808 06:03:51.430979 1801 consensus_queue.cc:237] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } }
I20260808 06:03:51.434351 1309 catalog_manager.cc:5495] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d reported cstate change: term changed from 0 to 1, leader changed from <none> to f022a886e7b1435eb8f7cb9cedb5996d (127.13.229.66). New cstate: current_term: 1 leader_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f022a886e7b1435eb8f7cb9cedb5996d" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 } health_report { overall_health: UNKNOWN } } }
I20260808 06:03:51.457432 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.453744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.13.229.66:35599)
0808 06:03:51.453886 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:51.453890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:51.453906 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:03:51.454017 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:51.454019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:51.454078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:51.454168 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:51.454437 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.454967 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.455870 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:51.456113 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:51.456199 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:51.456521 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:51.456975 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:51.457092 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:51.457191 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 06:03:51.457263 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:03:51.466187 1727 raft_consensus.cc:1261] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 1 FOLLOWER]: Refusing update from remote peer f022a886e7b1435eb8f7cb9cedb5996d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:51.466197 1459 raft_consensus.cc:1261] T d783751e1e2f4050896dd4e579fbfe58 P 3c14f988007b4d6b8c27ea8a47294235 [term 1 FOLLOWER]: Refusing update from remote peer f022a886e7b1435eb8f7cb9cedb5996d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:51.467185 1803 consensus_queue.cc:1035] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c14f988007b4d6b8c27ea8a47294235" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 41767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:51.467418 1801 consensus_queue.cc:1035] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e789b1e6c6404e8ccca611cc8f35f6" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:51.477129 1807 mvcc.cc:205] Tried to move back new op lower bound from 7316148352874987520 to 7316148352744022016. Current Snapshot: MvccSnapshot[applied={T|T < 7316148352874987520}]
I20260808 06:03:51.485105 1309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:51.485296 1309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:51.487911 1309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 516a280d3e2b47d9b294fc2c28ebf5fc: Sending DeleteTablet for 3 replicas of tablet d783751e1e2f4050896dd4e579fbfe58
I20260808 06:03:51.488745 1572 tablet_service.cc:1514] Processing DeleteTablet for tablet d783751e1e2f4050896dd4e579fbfe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:51 UTC) from {username='slave'} at 127.0.0.1:55590
I20260808 06:03:51.488931 1438 tablet_service.cc:1514] Processing DeleteTablet for tablet d783751e1e2f4050896dd4e579fbfe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:51 UTC) from {username='slave'} at 127.0.0.1:51606
I20260808 06:03:51.488947 1706 tablet_service.cc:1514] Processing DeleteTablet for tablet d783751e1e2f4050896dd4e579fbfe58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:51 UTC) from {username='slave'} at 127.0.0.1:47306
I20260808 06:03:51.489506 1815 tablet_replica.cc:330] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: stopping tablet replica
I20260808 06:03:51.489925 1815 raft_consensus.cc:2229] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:51.490053 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1374
I20260808 06:03:51.491300 1815 raft_consensus.cc:2258] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:51.492884 1817 tablet_replica.cc:330] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: stopping tablet replica
I20260808 06:03:51.493288 1815 ts_tablet_manager.cc:1905] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:51.493319 1817 raft_consensus.cc:2229] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:51.493755 1817 raft_consensus.cc:2258] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:51.495235 1817 ts_tablet_manager.cc:1905] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:51.496848 1815 ts_tablet_manager.cc:1918] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:51.496973 1815 log.cc:1196] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/d783751e1e2f4050896dd4e579fbfe58
I20260808 06:03:51.497334 1815 ts_tablet_manager.cc:1939] T d783751e1e2f4050896dd4e579fbfe58 P f022a886e7b1435eb8f7cb9cedb5996d: Deleting consensus metadata
I20260808 06:03:51.498663 1817 ts_tablet_manager.cc:1918] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:51.498737 1296 catalog_manager.cc:4841] TS f022a886e7b1435eb8f7cb9cedb5996d (127.13.229.66:35599): tablet d783751e1e2f4050896dd4e579fbfe58 (table test-table [id=492cff6c21d5418e923718fc46f75589]) successfully deleted
I20260808 06:03:51.498894 1817 log.cc:1196] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/d783751e1e2f4050896dd4e579fbfe58
I20260808 06:03:51.499251 1817 ts_tablet_manager.cc:1939] T d783751e1e2f4050896dd4e579fbfe58 P 00e789b1e6c6404e8ccca611cc8f35f6: Deleting consensus metadata
I20260808 06:03:51.500423 1300 catalog_manager.cc:4841] TS 00e789b1e6c6404e8ccca611cc8f35f6 (127.13.229.67:40585): tablet d783751e1e2f4050896dd4e579fbfe58 (table test-table [id=492cff6c21d5418e923718fc46f75589]) successfully deleted
W20260808 06:03:51.502967 1299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:41767 (error 108)
I20260808 06:03:51.503969 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1509
I20260808 06:03:51.504076 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:51.503739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:41767 (local address 127.0.0.1:51614)
0808 06:03:51.503884 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:51.503979 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 06:03:51.506371 1299 catalog_manager.cc:4584] TS 3c14f988007b4d6b8c27ea8a47294235 (127.13.229.65:41767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d783751e1e2f4050896dd4e579fbfe58: Network error: Client connection negotiation failed: client connection to 127.13.229.65:41767: connect: Connection refused (error 111)
I20260808 06:03:51.513389 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1643
I20260808 06:03:51.520344 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1281
2026-08-08T06:03:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1124 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:51 dist-test-slave-zkpd krb5kdc[1821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:51 dist-test-slave-zkpd krb5kdc[1821](info): set up 2 sockets
Aug 08 06:03:51 dist-test-slave-zkpd krb5kdc[1821](info): commencing operation
krb5kdc: starting...
W20260808 06:03:52.593617 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:52 dist-test-slave-zkpd krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:03:52.718863 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:32773
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:32773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:52.831884 1834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:52.832242 1834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:52.832319 1834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:52.836335 1834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:52.836432 1834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:52.836473 1834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:52.836534 1834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:52.836575 1834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:52.841835 1834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:32773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:32773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:52.842904 1834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:52.844018 1834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:52.852566 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:52.852632 1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:52.852658 1834 server_base.cc:1034] running on GCE node
W20260808 06:03:52.852577 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:52.853346 1834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:52.854425 1834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:52.855589 1834 hybrid_clock.cc:648] HybridClock initialized: now 1786169032855576 us; error 24 us; skew 500 ppm
Aug 08 06:03:52 dist-test-slave-zkpd krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:52.858306 1834 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:03:52.859102 1834 webserver.cc:460] Webserver started at http://127.13.229.126:46023/ using document root <none> and password file <none>
I20260808 06:03:52.859601 1834 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:52.859658 1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:52.859815 1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:52.861810 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "4fda8ace4ce44490967a32a0ef7a7c78"
format_stamp: "Formatted at 2026-08-08 06:03:52 on dist-test-slave-zkpd"
server_key: "77028c8d7740499eab58921795d8bdb0"
server_key_iv: "a4f0e5ea6051544e5af8a216c492eec9"
server_key_version: "encryptionkey@0"
I20260808 06:03:52.862326 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "4fda8ace4ce44490967a32a0ef7a7c78"
format_stamp: "Formatted at 2026-08-08 06:03:52 on dist-test-slave-zkpd"
server_key: "77028c8d7740499eab58921795d8bdb0"
server_key_iv: "a4f0e5ea6051544e5af8a216c492eec9"
server_key_version: "encryptionkey@0"
I20260808 06:03:52.865849 1834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:03:52.868147 1849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:52.869333 1834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:52.869457 1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "4fda8ace4ce44490967a32a0ef7a7c78"
format_stamp: "Formatted at 2026-08-08 06:03:52 on dist-test-slave-zkpd"
server_key: "77028c8d7740499eab58921795d8bdb0"
server_key_iv: "a4f0e5ea6051544e5af8a216c492eec9"
server_key_version: "encryptionkey@0"
I20260808 06:03:52.869556 1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:52.897490 1834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:52.900616 1834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:52.900816 1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:52.908130 1834 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:32773
I20260808 06:03:52.908157 1904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:32773 every 8 connection(s)
I20260808 06:03:52.909101 1834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:52.911974 1905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:52.915393 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1834
I20260808 06:03:52.915539 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:52.915815 14229 external_mini_cluster.cc:1428] Setting key 5d28a6a75d6a63b48172b83dbff2979a
I20260808 06:03:52.918251 1905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: Bootstrap starting.
I20260808 06:03:52.920588 1905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:52.921341 1905 log.cc:824] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:52.923213 1905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: No bootstrap required, opened a new log
Aug 08 06:03:52 dist-test-slave-zkpd krb5kdc[1821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:52.926699 1905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } }
I20260808 06:03:52.926954 1905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:52.927037 1905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda8ace4ce44490967a32a0ef7a7c78, State: Initialized, Role: FOLLOWER
I20260808 06:03:52.927512 1905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } }
I20260808 06:03:52.927665 1905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:52.927747 1905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:52.927858 1905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:52.929023 1905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } }
I20260808 06:03:52.929387 1905 leader_election.cc:302] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fda8ace4ce44490967a32a0ef7a7c78; no voters:
I20260808 06:03:52.929714 1905 leader_election.cc:288] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:52.929826 1910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:52.930090 1910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [term 1 LEADER]: Becoming Leader. State: Replica: 4fda8ace4ce44490967a32a0ef7a7c78, State: Running, Role: LEADER
I20260808 06:03:52.930401 1910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } }
I20260808 06:03:52.930598 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:52.917355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.13.229.126:32773)
0808 06:03:52.917826 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:52.917842 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:52.917881 (+ 39us) server_negotiation.cc:407] Connection header received
0808 06:03:52.918568 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:52.918588 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:52.918926 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:52.919262 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:52.920152 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:52.921228 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:52.922162 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:52.922653 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:52.922823 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:52.925388 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:52.925393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:52.925406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:52.925432 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:52.927299 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:52.927792 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:52.927799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:52.927805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:52.927981 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:52.928276 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:52.928280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:52.928282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:52.928807 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:52.929029 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:52.929224 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 06:03:52.929538 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260808 06:03:52.930989 1905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:52.932243 1911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } } }
I20260808 06:03:52.932399 1911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:52.932242 1912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fda8ace4ce44490967a32a0ef7a7c78. Latest consensus state: current_term: 1 leader_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fda8ace4ce44490967a32a0ef7a7c78" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 32773 } } }
I20260808 06:03:52.932505 1912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:52.933218 1920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:52.936565 1920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:52.942214 1920 catalog_manager.cc:1347] Generated new cluster ID: fd98bb2ca45444be84079f5525ae8bbb
I20260808 06:03:52.942286 1920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:52.952098 1920 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:52.953274 1920 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:52.962242 1920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: Generated new TSK 0
I20260808 06:03:52.963037 1920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.229.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
Entry for principal kudu/127.13.229.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab.
I20260808 06:03:53.003028 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:53.118444 1932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:53.118697 1932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:53.118742 1932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:53.122717 1932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:53.122776 1932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:53.122831 1932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:53.127883 1932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.65.keytab
--principal=kudu/127.13.229.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:53.128898 1932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:53.129951 1932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:53.139883 1938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.139986 1940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.140092 1932 server_base.cc:1034] running on GCE node
W20260808 06:03:53.140141 1937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.140564 1932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:53.141182 1932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:53.142385 1932 hybrid_clock.cc:648] HybridClock initialized: now 1786169033142361 us; error 30 us; skew 500 ppm
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:53.145385 1932 init.cc:377] Logged in from keytab as kudu/127.13.229.65@KRBTEST.COM (short username kudu)
I20260808 06:03:53.146237 1932 webserver.cc:460] Webserver started at http://127.13.229.65:38007/ using document root <none> and password file <none>
I20260808 06:03:53.146800 1932 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:53.146915 1932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:53.147145 1932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:53.149264 1932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "8f44c80c03ba4cac85d18193a3a19cd3"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "6c67000b108828752bbaa8e0b8f7ae13"
server_key_iv: "da13fbbc3121affe8650b5fe1a9d56c1"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.149840 1932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "8f44c80c03ba4cac85d18193a3a19cd3"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "6c67000b108828752bbaa8e0b8f7ae13"
server_key_iv: "da13fbbc3121affe8650b5fe1a9d56c1"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.153889 1932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 06:03:53.157060 1947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.158388 1932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:53.158489 1932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "8f44c80c03ba4cac85d18193a3a19cd3"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "6c67000b108828752bbaa8e0b8f7ae13"
server_key_iv: "da13fbbc3121affe8650b5fe1a9d56c1"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.158576 1932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:53.174188 1932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:53.181768 1932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:53.181943 1932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:53.182485 1932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:53.183292 1932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:53.183338 1932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.183378 1932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:53.183400 1932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.192622 1932 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:42901
I20260808 06:03:53.192653 2062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:42901 every 8 connection(s)
I20260808 06:03:53.193497 1932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:53.200505 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 1932
I20260808 06:03:53.200647 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:53.200969 14229 external_mini_cluster.cc:1428] Setting key 464d2a213aa2025f019082ca92dd8439
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.65@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:53.208043 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.195278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:43317 (local address 127.13.229.126:32773)
0808 06:03:53.195451 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.195456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.196296 (+ 840us) server_negotiation.cc:407] Connection header received
0808 06:03:53.197607 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.197610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.197661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.197738 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:53.198837 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.199429 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.200750 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.200993 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.201122 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.204084 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:53.204087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:53.204089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:53.204112 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:53.205652 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.206286 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.206292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.206297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.206402 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.206733 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.206739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.206744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.207013 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:53.207121 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.207616 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.207768 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 06:03:53.208693 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.195676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32773 (local address 127.13.229.65:43317)
0808 06:03:53.196150 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.196192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.197412 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.197909 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.197915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.198250 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:53.198712 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.198725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.199570 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.199573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.200528 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.200538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.201164 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.201168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.201348 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.201975 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:53.201991 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:53.203885 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:53.205775 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.205783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.205801 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.206161 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.206496 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.206499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.206500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.206627 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.207121 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:53.207126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:53.207346 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.207496 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.207735 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":168,"threads_started":1}
I20260808 06:03:53.209944 2063 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32773
I20260808 06:03:53.210170 2063 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:53.210649 2063 heartbeater.cc:511] Master 127.13.229.126:32773 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.229.66@KRBTEST.COM; defaulting to no policy
I20260808 06:03:53.212239 1867 ts_manager.cc:194] Registered new tserver with Master: 8f44c80c03ba4cac85d18193a3a19cd3 (127.13.229.65:42901)
I20260808 06:03:53.213402 1867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.65@KRBTEST.COM'} at 127.13.229.65:43317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
Entry for principal kudu/127.13.229.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab.
I20260808 06:03:53.230384 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:53.350966 2068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:53.351337 2068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:53.351415 2068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:53.358107 2068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:53.358163 2068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:53.358264 2068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:53.366387 2068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.66.keytab
--principal=kudu/127.13.229.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:53.367740 2068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:53.369154 2068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:53.379199 2073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.379206 2074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.379221 2076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.379464 2068 server_base.cc:1034] running on GCE node
I20260808 06:03:53.380064 2068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:53.380674 2068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:53.381844 2068 hybrid_clock.cc:648] HybridClock initialized: now 1786169033381812 us; error 45 us; skew 500 ppm
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:53.384550 2068 init.cc:377] Logged in from keytab as kudu/127.13.229.66@KRBTEST.COM (short username kudu)
I20260808 06:03:53.385401 2068 webserver.cc:460] Webserver started at http://127.13.229.66:46235/ using document root <none> and password file <none>
I20260808 06:03:53.385916 2068 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:53.385974 2068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:53.386142 2068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:53.388307 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "7cc73a7fd18b41bb900f855e2bf46ad1"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "5aa9b26679ff6819df906c1e6a66ef76"
server_key_iv: "073fa3b5f2247530201b72c7729f3d2b"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.388885 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "7cc73a7fd18b41bb900f855e2bf46ad1"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "5aa9b26679ff6819df906c1e6a66ef76"
server_key_iv: "073fa3b5f2247530201b72c7729f3d2b"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.392947 2068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 06:03:53.395325 2084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.396725 2068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:03:53.396965 2068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "7cc73a7fd18b41bb900f855e2bf46ad1"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "5aa9b26679ff6819df906c1e6a66ef76"
server_key_iv: "073fa3b5f2247530201b72c7729f3d2b"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.397094 2068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:53.412636 2068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:53.415403 2068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:53.415604 2068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:53.416270 2068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:53.417199 2068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:53.417246 2068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.417315 2068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:53.417362 2068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.426379 2068 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:35199
I20260808 06:03:53.426400 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:35199 every 8 connection(s)
I20260808 06:03:53.427331 2068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.66@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:53.437278 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2068
I20260808 06:03:53.437423 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:53.437721 14229 external_mini_cluster.cc:1428] Setting key 7083984c53d54233f5ba4634404cc55c
I20260808 06:03:53.441874 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.429252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:33347 (local address 127.13.229.126:32773)
0808 06:03:53.429440 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.429444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.430195 (+ 751us) server_negotiation.cc:407] Connection header received
0808 06:03:53.431689 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.431693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.431753 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.431873 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:53.433252 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.433852 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.434972 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.435183 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.435259 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.437893 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:53.437897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:53.437900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:53.437928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:53.439622 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.440213 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.440219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.440223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.440318 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.440667 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.440673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.440677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.440924 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:53.441026 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.441509 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.441655 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 06:03:53.442945 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.429569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32773 (local address 127.13.229.66:33347)
0808 06:03:53.430042 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.430086 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.431461 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.432130 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.432139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.432564 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:53.433092 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.433107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.433975 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.433978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.434864 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.434875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.435313 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.435315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.435475 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.436108 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:53.436125 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:53.437685 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:53.439751 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.439756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.439767 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.440116 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.440436 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.440438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.440440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.440569 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.441024 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:53.441029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:53.441249 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.441411 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.441808 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260808 06:03:53.444314 2203 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32773
I20260808 06:03:53.444593 2203 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:53.445091 2203 heartbeater.cc:511] Master 127.13.229.126:32773 requested a full tablet report, sending...
I20260808 06:03:53.446177 1867 ts_manager.cc:194] Registered new tserver with Master: 7cc73a7fd18b41bb900f855e2bf46ad1 (127.13.229.66:35199)
I20260808 06:03:53.446743 1867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.66@KRBTEST.COM'} at 127.13.229.66:33347
WARNING: no policy specified for kudu/127.13.229.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
Entry for principal kudu/127.13.229.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab.
I20260808 06:03:53.466765 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:53.578816 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:53.579063 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:53.579140 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:53.583091 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:53.583146 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:53.583240 2208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:53.588380 2208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.67.keytab
--principal=kudu/127.13.229.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:32773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:53.589394 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:53.590409 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:53.599840 2214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.599857 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.599839 2216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.600239 2208 server_base.cc:1034] running on GCE node
I20260808 06:03:53.600709 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:53.601326 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:53.602486 2208 hybrid_clock.cc:648] HybridClock initialized: now 1786169033602466 us; error 37 us; skew 500 ppm
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:53.605139 2208 init.cc:377] Logged in from keytab as kudu/127.13.229.67@KRBTEST.COM (short username kudu)
I20260808 06:03:53.605999 2208 webserver.cc:460] Webserver started at http://127.13.229.67:41235/ using document root <none> and password file <none>
I20260808 06:03:53.606551 2208 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:53.606613 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:53.606841 2208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:53.608875 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "82c30a5d81f24ce0af1e891746e233c8"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "7ab66a0267a8dafe586cbb7951c9d9cf"
server_key_iv: "aa4a05c6be4ffc573f02b3ddff9a6b3e"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.609436 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "82c30a5d81f24ce0af1e891746e233c8"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "7ab66a0267a8dafe586cbb7951c9d9cf"
server_key_iv: "aa4a05c6be4ffc573f02b3ddff9a6b3e"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.612949 2208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:03:53.615238 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.616462 2208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:53.616592 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "82c30a5d81f24ce0af1e891746e233c8"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "7ab66a0267a8dafe586cbb7951c9d9cf"
server_key_iv: "aa4a05c6be4ffc573f02b3ddff9a6b3e"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.616698 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:53.627506 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:53.630101 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:53.630296 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:53.630976 2208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:53.631829 2208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:53.631901 2208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.632052 2208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:53.632105 2208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.641220 2208 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:33613
I20260808 06:03:53.641250 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:33613 every 8 connection(s)
I20260808 06:03:53.642064 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:53.643563 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2208
I20260808 06:03:53.643718 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:53.644116 14229 external_mini_cluster.cc:1428] Setting key 509c40284d82f0d4724691537be3f3e5
Aug 08 06:03:53 dist-test-slave-zkpd krb5kdc[1821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169033, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.67@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:03:53.655727 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.643998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:58559 (local address 127.13.229.126:32773)
0808 06:03:53.644164 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.644170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.645169 (+ 999us) server_negotiation.cc:407] Connection header received
0808 06:03:53.646808 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.646811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.646883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.646966 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:53.648111 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.648633 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.649839 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.650050 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.650127 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.652638 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:53.652640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:53.652642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:53.652664 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:53.653987 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.654460 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.654462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.654464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.654535 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.654831 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.654833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.654834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.654995 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:53.655067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.655512 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.655607 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 06:03:53.656587 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.644493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:32773 (local address 127.13.229.67:58559)
0808 06:03:53.645021 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.645074 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.646600 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.647153 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.647160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.647499 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:53.647986 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.648001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.648739 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.648742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.649602 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.649611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.650177 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.650180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.650317 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.650969 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:03:53.650985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:53.652484 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:53.654095 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.654100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.654111 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.654365 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.654612 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:03:53.654614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:03:53.654615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:03:53.654745 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:03:53.655080 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:53.655085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:53.655307 (+ 222us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.655451 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.655688 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":171,"threads_started":1}
I20260808 06:03:53.657819 2339 heartbeater.cc:347] Connected to a master server at 127.13.229.126:32773
I20260808 06:03:53.658072 2339 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:53.658653 2339 heartbeater.cc:511] Master 127.13.229.126:32773 requested a full tablet report, sending...
I20260808 06:03:53.659677 1867 ts_manager.cc:194] Registered new tserver with Master: 82c30a5d81f24ce0af1e891746e233c8 (127.13.229.67:33613)
I20260808 06:03:53.660293 1867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.229.67@KRBTEST.COM'} at 127.13.229.67:58559
I20260808 06:03:53.670282 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:53.679077 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.672220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.13.229.126:32773)
0808 06:03:53.672344 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.672348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.672556 (+ 208us) server_negotiation.cc:407] Connection header received
0808 06:03:53.672676 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.672681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.672734 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.672794 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:53.673464 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.674013 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.674812 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.675015 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.675087 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.676472 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:53.676475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:53.676477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:53.676495 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:53.677668 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.678043 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.678045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.678046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.678115 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:53.678396 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:53.678398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:53.678399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:53.678589 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:53.678689 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.678882 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.678968 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:03:53.681695 1867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:53.684798 1867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:53.699097 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.693030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33613 (local address 127.0.0.1:49660)
0808 06:03:53.694172 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.694184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.694249 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.695066 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.695069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.695092 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:53.695194 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.695200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696458 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.696462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.697573 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.697583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.698539 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.698543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.698653 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.698694 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.698813 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.698935 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":106,"threads_started":1}
I20260808 06:03:53.699136 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.693756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.13.229.67:33613)
0808 06:03:53.694352 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.694366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.694390 (+ 24us) server_negotiation.cc:407] Connection header received
0808 06:03:53.694513 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.694520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.694727 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.694890 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:53.695328 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696171 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.697703 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.698405 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.698517 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.698662 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.698861 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.698978 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":148,"threads_started":1}
I20260808 06:03:53.699832 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.693333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:35199 (local address 127.0.0.1:34690)
0808 06:03:53.694478 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.694491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.694579 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.695119 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.695121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.695134 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:53.695199 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.695204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696615 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.696618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.698030 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.698041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.699391 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.699395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.699513 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.699533 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.699619 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.699698 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":119,"thread_start_us":116,"threads_started":1}
I20260808 06:03:53.700035 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.693797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.13.229.66:35199)
0808 06:03:53.694270 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.694286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.694677 (+ 391us) server_negotiation.cc:407] Connection header received
0808 06:03:53.694748 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.694751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.694983 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.695150 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:53.695302 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696452 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.698170 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.699024 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.699127 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.699316 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.699667 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.699763 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":109,"threads_started":1}
I20260808 06:03:53.702281 2273 tablet_service.cc:1467] Processing CreateTablet for tablet db297f3e3de040b89d92091c7f70443e (DEFAULT_TABLE table=test-table [id=24601b39baae4f68afb87587f0df1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:53.702497 2134 tablet_service.cc:1467] Processing CreateTablet for tablet db297f3e3de040b89d92091c7f70443e (DEFAULT_TABLE table=test-table [id=24601b39baae4f68afb87587f0df1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:53.703198 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db297f3e3de040b89d92091c7f70443e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:53.703464 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db297f3e3de040b89d92091c7f70443e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:53.708498 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.692958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42901 (local address 127.0.0.1:55364)
0808 06:03:53.693582 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.694092 (+ 510us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.694225 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.694888 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.694893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.694916 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:53.695054 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.695065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696615 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.696618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.698030 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.698041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.708054 (+ 10013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.708058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.708162 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.708189 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.708266 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.708347 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":167,"thread_start_us":53,"threads_started":1}
I20260808 06:03:53.709125 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.693105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.13.229.65:42901)
0808 06:03:53.694468 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.694482 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.694503 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:03:53.694569 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.694573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.694727 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.694890 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:53.695197 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.696461 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.698197 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.698990 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.699089 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.708797 (+ 9708us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.708860 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.708967 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":71,"threads_started":1}
I20260808 06:03:53.710284 2357 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Bootstrap starting.
I20260808 06:03:53.709728 2356 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Bootstrap starting.
I20260808 06:03:53.711637 1995 tablet_service.cc:1467] Processing CreateTablet for tablet db297f3e3de040b89d92091c7f70443e (DEFAULT_TABLE table=test-table [id=24601b39baae4f68afb87587f0df1865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:53.712517 2356 tablet_bootstrap.cc:654] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:53.712674 1995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db297f3e3de040b89d92091c7f70443e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:53.713961 2356 log.cc:824] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:53.714509 2357 tablet_bootstrap.cc:654] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:53.715651 2357 log.cc:824] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:53.716436 2356 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: No bootstrap required, opened a new log
I20260808 06:03:53.716641 2356 ts_tablet_manager.cc:1397] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 06:03:53.717384 2357 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: No bootstrap required, opened a new log
I20260808 06:03:53.717586 2357 ts_tablet_manager.cc:1397] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260808 06:03:53.719973 2360 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: Bootstrap starting.
I20260808 06:03:53.720654 2356 raft_consensus.cc:348] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.720650 2357 raft_consensus.cc:348] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.720953 2357 raft_consensus.cc:374] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:53.720954 2356 raft_consensus.cc:374] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:53.721009 2357 raft_consensus.cc:729] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c30a5d81f24ce0af1e891746e233c8, State: Initialized, Role: FOLLOWER
I20260808 06:03:53.721025 2356 raft_consensus.cc:729] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc73a7fd18b41bb900f855e2bf46ad1, State: Initialized, Role: FOLLOWER
I20260808 06:03:53.721481 2356 consensus_queue.cc:260] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.721539 2357 consensus_queue.cc:260] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.722229 2339 heartbeater.cc:503] Master 127.13.229.126:32773 was elected leader, sending a full tablet report...
I20260808 06:03:53.722458 2203 heartbeater.cc:503] Master 127.13.229.126:32773 was elected leader, sending a full tablet report...
I20260808 06:03:53.722544 2356 ts_tablet_manager.cc:1428] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 06:03:53.722764 2360 tablet_bootstrap.cc:654] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:53.723619 2360 log.cc:824] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:53.723779 2357 ts_tablet_manager.cc:1428] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 06:03:53.725783 2360 tablet_bootstrap.cc:492] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: No bootstrap required, opened a new log
I20260808 06:03:53.726003 2360 ts_tablet_manager.cc:1397] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:53.729671 2360 raft_consensus.cc:348] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.729954 2360 raft_consensus.cc:374] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:53.730024 2360 raft_consensus.cc:729] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f44c80c03ba4cac85d18193a3a19cd3, State: Initialized, Role: FOLLOWER
I20260808 06:03:53.730502 2360 consensus_queue.cc:260] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.730657 2361 raft_consensus.cc:482] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:53.730835 2361 raft_consensus.cc:504] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.731187 2063 heartbeater.cc:503] Master 127.13.229.126:32773 was elected leader, sending a full tablet report...
I20260808 06:03:53.731549 2360 ts_tablet_manager.cc:1428] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 06:03:53.732106 2361 leader_election.cc:288] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f44c80c03ba4cac85d18193a3a19cd3 (127.13.229.65:42901), 82c30a5d81f24ce0af1e891746e233c8 (127.13.229.67:33613)
I20260808 06:03:53.735833 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.732524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:45465 (local address 127.13.229.65:42901)
0808 06:03:53.732630 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.732635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.732651 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:03:53.732706 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.732709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.732751 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.732836 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:53.733139 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.733706 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.734986 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.735361 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.735451 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.735489 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.735702 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.735749 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 06:03:53.735881 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.732277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42901 (local address 127.13.229.66:45465)
0808 06:03:53.732390 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.732408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.732496 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.732847 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.732850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.732868 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:53.733012 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.733018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.733851 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.733854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.734857 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.734867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.735477 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.735480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.735585 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.735611 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.735694 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.735756 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260808 06:03:53.736485 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db297f3e3de040b89d92091c7f70443e" candidate_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" is_pre_election: true
I20260808 06:03:53.736727 2017 raft_consensus.cc:2454] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc73a7fd18b41bb900f855e2bf46ad1 in term 0.
I20260808 06:03:53.737248 2088 leader_election.cc:302] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc73a7fd18b41bb900f855e2bf46ad1, 8f44c80c03ba4cac85d18193a3a19cd3; no voters:
I20260808 06:03:53.737488 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.732328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:33613 (local address 127.13.229.66:39997)
0808 06:03:53.733131 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.733144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:53.733244 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:53.733851 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:53.733853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:53.733872 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:53.734010 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.734017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.734960 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.734966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.735971 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:53.735983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.737064 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.737067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:53.737178 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.737203 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 06:03:53.737286 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:03:53.737358 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20260808 06:03:53.737550 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.732677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:39997 (local address 127.13.229.67:33613)
0808 06:03:53.732808 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.732812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.733649 (+ 837us) server_negotiation.cc:407] Connection header received
0808 06:03:53.733693 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.733695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.733747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.733844 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:53.734131 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.734845 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.736213 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.736817 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.736953 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.736999 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.737307 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.737403 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:03:53.737769 2361 raft_consensus.cc:2781] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:53.737895 2361 raft_consensus.cc:482] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:53.737970 2361 raft_consensus.cc:3037] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:53.738080 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db297f3e3de040b89d92091c7f70443e" candidate_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c30a5d81f24ce0af1e891746e233c8" is_pre_election: true
I20260808 06:03:53.738341 2293 raft_consensus.cc:2454] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc73a7fd18b41bb900f855e2bf46ad1 in term 0.
I20260808 06:03:53.739151 2361 raft_consensus.cc:504] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.739456 2361 leader_election.cc:288] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f44c80c03ba4cac85d18193a3a19cd3 (127.13.229.65:42901), 82c30a5d81f24ce0af1e891746e233c8 (127.13.229.67:33613)
I20260808 06:03:53.739748 2017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db297f3e3de040b89d92091c7f70443e" candidate_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f44c80c03ba4cac85d18193a3a19cd3"
I20260808 06:03:53.739748 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db297f3e3de040b89d92091c7f70443e" candidate_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c30a5d81f24ce0af1e891746e233c8"
I20260808 06:03:53.739871 2017 raft_consensus.cc:3037] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:53.739883 2293 raft_consensus.cc:3037] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:53.741070 2017 raft_consensus.cc:2454] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc73a7fd18b41bb900f855e2bf46ad1 in term 1.
I20260808 06:03:53.741147 2293 raft_consensus.cc:2454] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc73a7fd18b41bb900f855e2bf46ad1 in term 1.
I20260808 06:03:53.741420 2088 leader_election.cc:302] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc73a7fd18b41bb900f855e2bf46ad1, 8f44c80c03ba4cac85d18193a3a19cd3; no voters:
I20260808 06:03:53.741611 2361 raft_consensus.cc:2781] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:53.741827 2361 raft_consensus.cc:686] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 7cc73a7fd18b41bb900f855e2bf46ad1, State: Running, Role: LEADER
I20260808 06:03:53.742146 2361 consensus_queue.cc:237] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } }
I20260808 06:03:53.745131 1866 catalog_manager.cc:5495] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cc73a7fd18b41bb900f855e2bf46ad1 (127.13.229.66). New cstate: current_term: 1 leader_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cc73a7fd18b41bb900f855e2bf46ad1" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 35199 } health_report { overall_health: HEALTHY } } }
I20260808 06:03:53.760607 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.757308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.13.229.66:35199)
0808 06:03:53.757434 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:53.757438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:53.757451 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:03:53.757584 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:53.757587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:53.757638 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:53.757722 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:53.757936 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.758419 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.759210 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:53.759428 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:53.759508 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:53.759755 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:53.760208 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:53.760299 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:53.760359 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 06:03:53.760413 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 06:03:53.766430 2017 raft_consensus.cc:1261] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3 [term 1 FOLLOWER]: Refusing update from remote peer 7cc73a7fd18b41bb900f855e2bf46ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:53.766429 2293 raft_consensus.cc:1261] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 1 FOLLOWER]: Refusing update from remote peer 7cc73a7fd18b41bb900f855e2bf46ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:53.767122 2366 consensus_queue.cc:1035] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c30a5d81f24ce0af1e891746e233c8" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:53.767314 2361 consensus_queue.cc:1035] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f44c80c03ba4cac85d18193a3a19cd3" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:53.775430 2369 mvcc.cc:205] Tried to move back new op lower bound from 7316148362300043264 to 7316148362210295808. Current Snapshot: MvccSnapshot[applied={T|T < 7316148362300043264}]
I20260808 06:03:53.777289 2370 mvcc.cc:205] Tried to move back new op lower bound from 7316148362300043264 to 7316148362210295808. Current Snapshot: MvccSnapshot[applied={T|T < 7316148362300043264}]
I20260808 06:03:53.784468 1866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:53.784622 1866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:53.787070 1866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fda8ace4ce44490967a32a0ef7a7c78: Sending DeleteTablet for 3 replicas of tablet db297f3e3de040b89d92091c7f70443e
I20260808 06:03:53.787878 1995 tablet_service.cc:1514] Processing DeleteTablet for tablet db297f3e3de040b89d92091c7f70443e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:55364
I20260808 06:03:53.787909 2273 tablet_service.cc:1514] Processing DeleteTablet for tablet db297f3e3de040b89d92091c7f70443e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:49660
I20260808 06:03:53.787958 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet db297f3e3de040b89d92091c7f70443e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:53 UTC) from {username='kudu', principal='kudu/127.13.229.126@KRBTEST.COM'} at 127.0.0.1:34690
I20260808 06:03:53.788460 2381 tablet_replica.cc:330] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: stopping tablet replica
I20260808 06:03:53.788782 2380 tablet_replica.cc:330] T db297f3e3de040b89d92091c7f70443e P 8f44c80c03ba4cac85d18193a3a19cd3: stopping tablet replica
I20260808 06:03:53.788789 2381 raft_consensus.cc:2229] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:53.788889 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1932
I20260808 06:03:53.790478 2381 raft_consensus.cc:2258] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:53.790939 2379 tablet_replica.cc:330] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: stopping tablet replica
I20260808 06:03:53.791211 2379 raft_consensus.cc:2229] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:53.791592 2379 raft_consensus.cc:2258] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:53.792528 2381 ts_tablet_manager.cc:1905] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:53.792939 2379 ts_tablet_manager.cc:1905] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:53.795496 2381 ts_tablet_manager.cc:1918] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:53.795612 2381 log.cc:1196] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/db297f3e3de040b89d92091c7f70443e
I20260808 06:03:53.795673 2379 ts_tablet_manager.cc:1918] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:53.795773 2379 log.cc:1196] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/db297f3e3de040b89d92091c7f70443e
I20260808 06:03:53.796021 2381 ts_tablet_manager.cc:1939] T db297f3e3de040b89d92091c7f70443e P 82c30a5d81f24ce0af1e891746e233c8: Deleting consensus metadata
I20260808 06:03:53.796169 2379 ts_tablet_manager.cc:1939] T db297f3e3de040b89d92091c7f70443e P 7cc73a7fd18b41bb900f855e2bf46ad1: Deleting consensus metadata
I20260808 06:03:53.797268 1851 catalog_manager.cc:4841] TS 7cc73a7fd18b41bb900f855e2bf46ad1 (127.13.229.66:35199): tablet db297f3e3de040b89d92091c7f70443e (table test-table [id=24601b39baae4f68afb87587f0df1865]) successfully deleted
I20260808 06:03:53.797430 1850 catalog_manager.cc:4841] TS 82c30a5d81f24ce0af1e891746e233c8 (127.13.229.67:33613): tablet db297f3e3de040b89d92091c7f70443e (table test-table [id=24601b39baae4f68afb87587f0df1865]) successfully deleted
W20260808 06:03:53.802896 1853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:42901 (error 108)
I20260808 06:03:53.803807 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2068
I20260808 06:03:53.804039 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:53.803648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:42901 (local address 127.0.0.1:55378)
0808 06:03:53.803798 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:53.803868 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 06:03:53.805034 1853 catalog_manager.cc:4584] TS 8f44c80c03ba4cac85d18193a3a19cd3 (127.13.229.65:42901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db297f3e3de040b89d92091c7f70443e: Network error: Client connection negotiation failed: client connection to 127.13.229.65:42901: connect: Connection refused (error 111)
I20260808 06:03:53.812021 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2208
I20260808 06:03:53.818848 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 1834
2026-08-08T06:03:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T06:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:53Z Disabled control of system clock
I20260808 06:03:53.851342 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42255
--webserver_interface=127.13.229.126
--webserver_port=0
--builtin_ntp_servers=127.13.229.84:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:53.963495 2386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:53.963745 2386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:53.963788 2386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:53.967826 2386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:53.967921 2386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:53.967955 2386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:53.967974 2386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:53.967989 2386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:53.972685 2386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:42255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:53.973702 2386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:53.975056 2386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:53.983443 2393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.983467 2386 server_base.cc:1034] running on GCE node
W20260808 06:03:53.983410 2395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:53.983426 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:53.984331 2386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:53.985426 2386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:53.986613 2386 hybrid_clock.cc:648] HybridClock initialized: now 1786169033986585 us; error 41 us; skew 500 ppm
I20260808 06:03:53.988564 2386 webserver.cc:460] Webserver started at http://127.13.229.126:44737/ using document root <none> and password file <none>
I20260808 06:03:53.989148 2386 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:53.989239 2386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:53.989475 2386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:53.991489 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "2c137366a59b4434bb138be31b411990"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "53fd3da5b1f6fd20461679a6abc1545b"
server_key_iv: "4ad493b1ea5dddcb88998b7b1c7b248c"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.992107 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "2c137366a59b4434bb138be31b411990"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "53fd3da5b1f6fd20461679a6abc1545b"
server_key_iv: "4ad493b1ea5dddcb88998b7b1c7b248c"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.995609 2386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 06:03:53.998059 2401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:53.999239 2386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:03:53.999395 2386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "2c137366a59b4434bb138be31b411990"
format_stamp: "Formatted at 2026-08-08 06:03:53 on dist-test-slave-zkpd"
server_key: "53fd3da5b1f6fd20461679a6abc1545b"
server_key_iv: "4ad493b1ea5dddcb88998b7b1c7b248c"
server_key_version: "encryptionkey@0"
I20260808 06:03:53.999511 2386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:54.018678 2386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:54.019385 2386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:54.019587 2386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:54.026717 2386 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:42255
I20260808 06:03:54.026724 2454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:42255 every 8 connection(s)
I20260808 06:03:54.027710 2386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:54.030790 2455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:54.036197 2455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: Bootstrap starting.
I20260808 06:03:54.037360 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2386
I20260808 06:03:54.037511 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:54.037781 14229 external_mini_cluster.cc:1428] Setting key 79d7178f9bdcd70a6c3c538c81eb7e71
I20260808 06:03:54.038766 2455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:54.039494 2455 log.cc:824] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:54.041436 2455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: No bootstrap required, opened a new log
I20260808 06:03:54.044940 2455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } }
I20260808 06:03:54.045169 2455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:54.045264 2455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c137366a59b4434bb138be31b411990, State: Initialized, Role: FOLLOWER
I20260808 06:03:54.045737 2455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } }
I20260808 06:03:54.045892 2455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:54.045959 2455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:54.046066 2455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:54.046871 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.038901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.13.229.126:42255)
0808 06:03:54.039596 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.039616 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.039666 (+ 50us) server_negotiation.cc:407] Connection header received
0808 06:03:54.040522 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.040543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.040863 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.041201 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:54.041669 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.042631 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.043525 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.044068 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.044251 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.045091 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:54.045104 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:54.045118 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:54.045590 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:54.045645 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:54.045665 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:54.045776 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.045892 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.046151 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260808 06:03:54.046890 2455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } }
I20260808 06:03:54.047237 2455 leader_election.cc:302] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c137366a59b4434bb138be31b411990; no voters:
I20260808 06:03:54.047560 2455 leader_election.cc:288] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:54.047655 2460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:54.048770 2455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:54.047824 2460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [term 1 LEADER]: Becoming Leader. State: Replica: 2c137366a59b4434bb138be31b411990, State: Running, Role: LEADER
I20260808 06:03:54.049160 2460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } }
I20260808 06:03:54.050948 2461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c137366a59b4434bb138be31b411990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } } }
I20260808 06:03:54.050972 2462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c137366a59b4434bb138be31b411990. Latest consensus state: current_term: 1 leader_uuid: "2c137366a59b4434bb138be31b411990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c137366a59b4434bb138be31b411990" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 42255 } } }
I20260808 06:03:54.051355 2461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:54.051358 2462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990 [sys.catalog]: This master's current role is: LEADER
W20260808 06:03:54.053718 2474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:03:54.053812 2474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:03:54.053892 2475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:54.054561 2475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:54.059904 2475 catalog_manager.cc:1347] Generated new cluster ID: f8f78e7d8b5845ccb6019f27a42bde60
I20260808 06:03:54.059967 2475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:54.067996 2475 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:54.068802 2475 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:54.078063 2475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: Generated new TSK 0
I20260808 06:03:54.078712 2475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:54.084175 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.65:0
--local_ip_for_outbound_sockets=127.13.229.65
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--builtin_ntp_servers=127.13.229.84:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 06:03:54.208997 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:54.209256 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:54.209295 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:54.213356 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:54.213413 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:54.213465 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.65
I20260808 06:03:54.218066 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.229.65
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.65
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:54.219189 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:54.220722 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:03:54.230755 2479 server_base.cc:1034] running on GCE node
W20260808 06:03:54.230718 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:54.230707 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:54.230707 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:54.231415 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:54.232067 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:54.233225 2479 hybrid_clock.cc:648] HybridClock initialized: now 1786169034233211 us; error 28 us; skew 500 ppm
I20260808 06:03:54.235018 2479 webserver.cc:460] Webserver started at http://127.13.229.65:44039/ using document root <none> and password file <none>
I20260808 06:03:54.235585 2479 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:54.235647 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:54.235939 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:54.237967 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data/instance:
uuid: "c4980ac04fca4a878304525f4aa5c5e7"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "72944f6a96696a5917156508c2aaaf46"
server_key_iv: "16c68a8448d85c2956135300541d348a"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.238538 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance:
uuid: "c4980ac04fca4a878304525f4aa5c5e7"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "72944f6a96696a5917156508c2aaaf46"
server_key_iv: "16c68a8448d85c2956135300541d348a"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.242156 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:03:54.244580 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.245755 2479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:54.245893 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal
uuid: "c4980ac04fca4a878304525f4aa5c5e7"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "72944f6a96696a5917156508c2aaaf46"
server_key_iv: "16c68a8448d85c2956135300541d348a"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.246003 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:54.271063 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:54.271811 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:54.272055 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:54.272718 2479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:54.273713 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:54.273808 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.273885 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:54.273921 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.283041 2479 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.65:36291
I20260808 06:03:54.283082 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.65:36291 every 8 connection(s)
I20260808 06:03:54.283991 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/data/info.pb
I20260808 06:03:54.292289 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2479
I20260808 06:03:54.292456 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-0/wal/instance
I20260808 06:03:54.292760 14229 external_mini_cluster.cc:1428] Setting key 58be6540bc4340733d3f4f22e880856c
I20260808 06:03:54.293041 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.285802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:40299 (local address 127.13.229.126:42255)
0808 06:03:54.285952 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.285957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.287064 (+ 1107us) server_negotiation.cc:407] Connection header received
0808 06:03:54.288512 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.288515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.288561 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.288635 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:54.289484 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.290019 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.291096 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.291316 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.291403 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.292540 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:54.292544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:54.292548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:54.292563 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:54.292588 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:54.292596 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:54.292673 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.292834 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.292898 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 06:03:54.294072 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.286271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42255 (local address 127.13.229.65:40299)
0808 06:03:54.286917 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.286959 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.288316 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.288793 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.288799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.288880 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:54.289363 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.289375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.290130 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.290132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.290994 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.291004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.291448 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.291450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.291587 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.292270 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:54.292272 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:54.292310 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:54.292369 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:54.292689 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:54.292698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:54.292707 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.292833 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.293116 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260808 06:03:54.294780 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.66:0
--local_ip_for_outbound_sockets=127.13.229.66
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--builtin_ntp_servers=127.13.229.84:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:54.295305 2611 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42255
I20260808 06:03:54.295552 2611 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:54.296175 2611 heartbeater.cc:511] Master 127.13.229.126:42255 requested a full tablet report, sending...
I20260808 06:03:54.297891 2419 ts_manager.cc:194] Registered new tserver with Master: c4980ac04fca4a878304525f4aa5c5e7 (127.13.229.65:36291)
I20260808 06:03:54.299055 2419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.65:40299
W20260808 06:03:54.407843 2614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:54.408138 2614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:54.408211 2614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:54.412120 2614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:54.412196 2614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:54.412285 2614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.66
I20260808 06:03:54.416956 2614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.229.66
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.66
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:54.418004 2614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:54.419366 2614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:54.428679 2620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:54.428690 2622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:54.428697 2619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:54.429023 2614 server_base.cc:1034] running on GCE node
I20260808 06:03:54.429451 2614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:54.430063 2614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:54.431257 2614 hybrid_clock.cc:648] HybridClock initialized: now 1786169034431239 us; error 34 us; skew 500 ppm
I20260808 06:03:54.433135 2614 webserver.cc:460] Webserver started at http://127.13.229.66:35007/ using document root <none> and password file <none>
I20260808 06:03:54.433713 2614 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:54.433822 2614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:54.434053 2614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:54.436214 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data/instance:
uuid: "a219c0580e574282b77d86bf337aec6e"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "3b30d91dc9f6dc468d74d1e50483d1ed"
server_key_iv: "ae83200e1975f0f3e3db160d6bfb4579"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.436807 2614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance:
uuid: "a219c0580e574282b77d86bf337aec6e"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "3b30d91dc9f6dc468d74d1e50483d1ed"
server_key_iv: "ae83200e1975f0f3e3db160d6bfb4579"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.440356 2614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 06:03:54.442685 2628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.443796 2614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:54.443953 2614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal
uuid: "a219c0580e574282b77d86bf337aec6e"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "3b30d91dc9f6dc468d74d1e50483d1ed"
server_key_iv: "ae83200e1975f0f3e3db160d6bfb4579"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.444067 2614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:54.459012 2614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:54.459656 2614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:54.459853 2614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:54.460465 2614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:54.461381 2614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:54.461462 2614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.461534 2614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:54.461586 2614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.470386 2614 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.66:40825
I20260808 06:03:54.470425 2744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.66:40825 every 8 connection(s)
I20260808 06:03:54.471246 2614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/data/info.pb
I20260808 06:03:54.472024 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2614
I20260808 06:03:54.472179 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal/instance
I20260808 06:03:54.472527 14229 external_mini_cluster.cc:1428] Setting key 111af337e3dcf66ca75efbcf2ea9fbc7
I20260808 06:03:54.474642 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.229.67:0
--local_ip_for_outbound_sockets=127.13.229.67
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--builtin_ntp_servers=127.13.229.84:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 06:03:54.486341 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.473345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.66:37623 (local address 127.13.229.126:42255)
0808 06:03:54.473492 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.473495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.474675 (+ 1180us) server_negotiation.cc:407] Connection header received
0808 06:03:54.476133 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.476139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.476204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.476304 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:54.477543 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.478240 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.479388 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.479810 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.479979 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.485724 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:54.485728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:54.485732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:54.485753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:54.485784 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:54.485799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:54.485895 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.486045 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.486131 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 06:03:54.487121 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.473747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42255 (local address 127.13.229.66:37623)
0808 06:03:54.474501 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.474552 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.475926 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.476540 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.476552 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.476668 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:54.477386 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.477408 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.478377 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.478380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.479248 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.479258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.480002 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.480006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.480146 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.485409 (+ 5263us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:54.485411 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:54.485456 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:54.485525 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:54.485896 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:54.485904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:54.485908 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.486007 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.486244 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
I20260808 06:03:54.488368 2745 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42255
I20260808 06:03:54.488631 2745 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:54.489178 2745 heartbeater.cc:511] Master 127.13.229.126:42255 requested a full tablet report, sending...
I20260808 06:03:54.490270 2419 ts_manager.cc:194] Registered new tserver with Master: a219c0580e574282b77d86bf337aec6e (127.13.229.66:40825)
I20260808 06:03:54.490846 2419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.66:37623
W20260808 06:03:54.589361 2748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:54.589612 2748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:54.589684 2748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:54.593608 2748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:54.593658 2748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:54.593765 2748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.229.67
I20260808 06:03:54.598394 2748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.229.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.229.67
--webserver_port=0
--tserver_master_addrs=127.13.229.126:42255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.229.67
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:54.599442 2748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:54.600818 2748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:54.610404 2754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:54.610389 2753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:54.610579 2748 server_base.cc:1034] running on GCE node
W20260808 06:03:54.610405 2756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:54.611094 2748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:54.611750 2748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:54.612910 2748 hybrid_clock.cc:648] HybridClock initialized: now 1786169034612877 us; error 45 us; skew 500 ppm
I20260808 06:03:54.614660 2748 webserver.cc:460] Webserver started at http://127.13.229.67:35567/ using document root <none> and password file <none>
I20260808 06:03:54.615216 2748 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:54.615276 2748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:54.615518 2748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:54.617539 2748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data/instance:
uuid: "a3804dd02f1542899d2eef002ea1f444"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "b5e50d0760a6c3667910fd6563d8a8c2"
server_key_iv: "62d1c5e1ca60812282a97f7c91e2cdee"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.618098 2748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance:
uuid: "a3804dd02f1542899d2eef002ea1f444"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "b5e50d0760a6c3667910fd6563d8a8c2"
server_key_iv: "62d1c5e1ca60812282a97f7c91e2cdee"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.621716 2748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:03:54.624084 2762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.625398 2748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:54.625531 2748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal
uuid: "a3804dd02f1542899d2eef002ea1f444"
format_stamp: "Formatted at 2026-08-08 06:03:54 on dist-test-slave-zkpd"
server_key: "b5e50d0760a6c3667910fd6563d8a8c2"
server_key_iv: "62d1c5e1ca60812282a97f7c91e2cdee"
server_key_version: "encryptionkey@0"
I20260808 06:03:54.625640 2748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:54.634936 2748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:54.635560 2748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:54.635742 2748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:54.636348 2748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 06:03:54.637238 2748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 06:03:54.637307 2748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.637378 2748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 06:03:54.637428 2748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:54.645982 2748 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.67:44187
I20260808 06:03:54.646008 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.67:44187 every 8 connection(s)
I20260808 06:03:54.646904 2748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/data/info.pb
I20260808 06:03:54.652292 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2748
I20260808 06:03:54.652439 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal/instance
I20260808 06:03:54.652736 14229 external_mini_cluster.cc:1428] Setting key 9fcf272d4a8ce94c533ad74f49f282e8
I20260808 06:03:54.657151 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.648828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.67:33401 (local address 127.13.229.126:42255)
0808 06:03:54.648974 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.648978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.650257 (+ 1279us) server_negotiation.cc:407] Connection header received
0808 06:03:54.651804 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.651806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.651854 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.651981 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:54.653164 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.653817 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.655270 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.655515 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.655597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.656741 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:54.656744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:54.656746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:54.656757 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:54.656774 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:54.656781 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:54.656837 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.656994 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.657043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:03:54.657984 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.649264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:42255 (local address 127.13.229.67:33401)
0808 06:03:54.650101 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.650160 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.651614 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.652184 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.652194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.652305 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:03:54.653003 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.653025 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.653957 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.653960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.655163 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.655173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.655658 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.655661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.655816 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.656508 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 06:03:54.656510 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:03:54.656550 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 06:03:54.656610 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:03:54.656855 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:03:54.656863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:03:54.656867 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.656966 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.657177 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":177,"threads_started":1}
I20260808 06:03:54.659020 2879 heartbeater.cc:347] Connected to a master server at 127.13.229.126:42255
I20260808 06:03:54.659240 2879 heartbeater.cc:464] Registering TS with master...
I20260808 06:03:54.659732 2879 heartbeater.cc:511] Master 127.13.229.126:42255 requested a full tablet report, sending...
I20260808 06:03:54.660727 2419 ts_manager.cc:194] Registered new tserver with Master: a3804dd02f1542899d2eef002ea1f444 (127.13.229.67:44187)
I20260808 06:03:54.661234 2419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.229.67:33401
I20260808 06:03:54.666591 14229 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 06:03:54.672047 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.668534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.13.229.126:42255)
0808 06:03:54.668646 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.668650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.668882 (+ 232us) server_negotiation.cc:407] Connection header received
0808 06:03:54.668979 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.668981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.669018 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.669083 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:54.669358 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.669836 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.670626 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.670822 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.670893 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.671628 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:54.671631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:54.671633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:54.671645 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:54.671661 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:54.671669 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:54.671725 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.671825 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.671871 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:03:54.674801 2419 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 06:03:54.677767 2419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 06:03:54.692377 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.687287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.13.229.67:44187)
0808 06:03:54.687771 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.687782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.688223 (+ 441us) server_negotiation.cc:407] Connection header received
0808 06:03:54.688308 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.688314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.688505 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.688649 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:54.688875 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.689732 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691144 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.691672 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.692005 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.692105 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.692203 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":160,"threads_started":1}
I20260808 06:03:54.692371 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.686067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:40825 (local address 127.0.0.1:52704)
0808 06:03:54.686442 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.686490 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.686613 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.687435 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.687441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.687466 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:54.687632 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.687640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.689266 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.689269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.690872 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.690884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.691737 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.691857 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.691944 (+ 87us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.692060 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.692202 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":177,"thread_start_us":108,"threads_started":1}
I20260808 06:03:54.692384 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.686857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:44187 (local address 127.0.0.1:55650)
0808 06:03:54.688093 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.688108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.688212 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.688651 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.688654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.688673 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:54.688752 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.688757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.690048 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.690051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.691021 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.691033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.691815 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.691966 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.691990 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.692073 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.692206 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"mutex_wait_us":59,"thread_start_us":107,"threads_started":1}
I20260808 06:03:54.693020 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.686233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.13.229.66:40825)
0808 06:03:54.686895 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.686909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.686937 (+ 28us) server_negotiation.cc:407] Connection header received
0808 06:03:54.687022 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.687030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.687254 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.687796 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:54.688000 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.689154 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691014 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.691606 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691691 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.691927 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.692737 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.692818 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":155,"threads_started":1}
I20260808 06:03:54.693681 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.686299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:36291 (local address 127.0.0.1:56020)
0808 06:03:54.686893 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.686907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.686999 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.687827 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.687830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.687846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:54.687963 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.687971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.689137 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.689143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.690872 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.690884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.693291 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.693295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.693394 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.693417 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.693480 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.693545 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":185,"thread_start_us":110,"threads_started":1}
I20260808 06:03:54.694217 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.686845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.13.229.65:36291)
0808 06:03:54.687286 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.687298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.687323 (+ 25us) server_negotiation.cc:407] Connection header received
0808 06:03:54.687406 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.687412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.687624 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.687788 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:54.688091 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.689031 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.691988 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.692785 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.692913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.693923 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.693976 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.694066 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":108,"threads_started":1}
I20260808 06:03:54.695580 2810 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 (DEFAULT_TABLE table=test-table [id=ff9520eb4cac48649860a4be6f4d6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:54.695583 2677 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 (DEFAULT_TABLE table=test-table [id=ff9520eb4cac48649860a4be6f4d6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:54.696614 2810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb7aef3634fdab75cbae422138a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:54.696640 2677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb7aef3634fdab75cbae422138a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:54.698457 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 (DEFAULT_TABLE table=test-table [id=ff9520eb4cac48649860a4be6f4d6d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 06:03:54.699209 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5bb7aef3634fdab75cbae422138a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:54.703166 2896 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Bootstrap starting.
I20260808 06:03:54.705394 2897 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Bootstrap starting.
I20260808 06:03:54.705668 2896 tablet_bootstrap.cc:654] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:54.706459 2896 log.cc:824] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:54.707556 2897 tablet_bootstrap.cc:654] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:54.708420 2897 log.cc:824] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:54.708860 2896 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: No bootstrap required, opened a new log
I20260808 06:03:54.709084 2896 ts_tablet_manager.cc:1397] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 06:03:54.709158 2899 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Bootstrap starting.
I20260808 06:03:54.710346 2897 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: No bootstrap required, opened a new log
I20260808 06:03:54.710534 2897 ts_tablet_manager.cc:1397] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 06:03:54.711236 2899 tablet_bootstrap.cc:654] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:54.712044 2899 log.cc:824] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:54.712576 2896 raft_consensus.cc:348] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.712852 2896 raft_consensus.cc:374] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:54.712920 2896 raft_consensus.cc:729] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3804dd02f1542899d2eef002ea1f444, State: Initialized, Role: FOLLOWER
I20260808 06:03:54.713374 2896 consensus_queue.cc:260] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.714000 2879 heartbeater.cc:503] Master 127.13.229.126:42255 was elected leader, sending a full tablet report...
I20260808 06:03:54.714031 2899 tablet_bootstrap.cc:492] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: No bootstrap required, opened a new log
I20260808 06:03:54.714195 2899 ts_tablet_manager.cc:1397] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 06:03:54.714768 2897 raft_consensus.cc:348] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.715013 2897 raft_consensus.cc:374] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:54.715062 2897 raft_consensus.cc:729] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4980ac04fca4a878304525f4aa5c5e7, State: Initialized, Role: FOLLOWER
I20260808 06:03:54.715516 2897 consensus_queue.cc:260] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.716202 2611 heartbeater.cc:503] Master 127.13.229.126:42255 was elected leader, sending a full tablet report...
I20260808 06:03:54.716550 2897 ts_tablet_manager.cc:1428] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 06:03:54.717331 2896 ts_tablet_manager.cc:1428] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260808 06:03:54.718277 2899 raft_consensus.cc:348] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.718490 2899 raft_consensus.cc:374] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:54.718554 2899 raft_consensus.cc:729] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a219c0580e574282b77d86bf337aec6e, State: Initialized, Role: FOLLOWER
I20260808 06:03:54.718989 2899 consensus_queue.cc:260] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.719616 2745 heartbeater.cc:503] Master 127.13.229.126:42255 was elected leader, sending a full tablet report...
I20260808 06:03:54.719854 2899 ts_tablet_manager.cc:1428] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260808 06:03:54.722210 2746 tablet.cc:2369] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 06:03:54.785161 2612 tablet.cc:2369] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:54.886178 2903 raft_consensus.cc:482] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 06:03:54.886438 2903 raft_consensus.cc:504] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.887656 2903 leader_election.cc:288] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a219c0580e574282b77d86bf337aec6e (127.13.229.66:40825), a3804dd02f1542899d2eef002ea1f444 (127.13.229.67:44187)
I20260808 06:03:54.891935 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.888132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:48113 (local address 127.13.229.66:40825)
0808 06:03:54.888301 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.888306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.888477 (+ 171us) server_negotiation.cc:407] Connection header received
0808 06:03:54.888652 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.888658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.888716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.888767 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:54.889265 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.889770 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.890949 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.891342 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.891482 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.891536 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.891705 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.891776 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 06:03:54.891965 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.888604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.229.65:41287 (local address 127.13.229.67:44187)
0808 06:03:54.888736 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.888743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.888760 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:03:54.888915 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.888919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.888973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.889049 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 06:03:54.889314 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.889838 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.890964 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.891365 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.891505 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.891553 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.891733 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.891824 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 06:03:54.891975 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.888013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.66:40825 (local address 127.13.229.65:48113)
0808 06:03:54.888323 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.888340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.888499 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.888971 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.888974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.888994 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:54.889134 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.889142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.889887 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.889891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.890808 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.890818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.891472 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.891475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.891565 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.891588 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.891686 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.891755 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":62,"thread_start_us":116,"threads_started":1}
I20260808 06:03:54.891985 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.888444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.67:44187 (local address 127.13.229.65:41287)
0808 06:03:54.888710 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.888722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:03:54.888797 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:03:54.889070 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:03:54.889072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:03:54.889086 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 06:03:54.889192 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.889196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.889961 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.889965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.890811 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:03:54.890820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.891501 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.891505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:03:54.891608 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.891626 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 06:03:54.891716 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 06:03:54.891788 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260808 06:03:54.892885 2830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5bb7aef3634fdab75cbae422138a93" candidate_uuid: "c4980ac04fca4a878304525f4aa5c5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3804dd02f1542899d2eef002ea1f444" is_pre_election: true
I20260808 06:03:54.892897 2698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5bb7aef3634fdab75cbae422138a93" candidate_uuid: "c4980ac04fca4a878304525f4aa5c5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a219c0580e574282b77d86bf337aec6e" is_pre_election: true
I20260808 06:03:54.893186 2830 raft_consensus.cc:2454] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4980ac04fca4a878304525f4aa5c5e7 in term 0.
I20260808 06:03:54.893203 2698 raft_consensus.cc:2454] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4980ac04fca4a878304525f4aa5c5e7 in term 0.
I20260808 06:03:54.893705 2497 leader_election.cc:302] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a219c0580e574282b77d86bf337aec6e, c4980ac04fca4a878304525f4aa5c5e7; no voters:
I20260808 06:03:54.894001 2903 raft_consensus.cc:2781] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 06:03:54.894098 2903 raft_consensus.cc:482] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 06:03:54.894176 2903 raft_consensus.cc:3037] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:54.895233 2903 raft_consensus.cc:504] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.895571 2903 leader_election.cc:288] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [CANDIDATE]: Term 1 election: Requested vote from peers a219c0580e574282b77d86bf337aec6e (127.13.229.66:40825), a3804dd02f1542899d2eef002ea1f444 (127.13.229.67:44187)
I20260808 06:03:54.895964 2698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5bb7aef3634fdab75cbae422138a93" candidate_uuid: "c4980ac04fca4a878304525f4aa5c5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a219c0580e574282b77d86bf337aec6e"
I20260808 06:03:54.895999 2830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5bb7aef3634fdab75cbae422138a93" candidate_uuid: "c4980ac04fca4a878304525f4aa5c5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3804dd02f1542899d2eef002ea1f444"
I20260808 06:03:54.896070 2698 raft_consensus.cc:3037] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:54.896138 2830 raft_consensus.cc:3037] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:54.897145 2830 raft_consensus.cc:2454] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4980ac04fca4a878304525f4aa5c5e7 in term 1.
I20260808 06:03:54.897152 2698 raft_consensus.cc:2454] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4980ac04fca4a878304525f4aa5c5e7 in term 1.
I20260808 06:03:54.897480 2497 leader_election.cc:302] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3804dd02f1542899d2eef002ea1f444, c4980ac04fca4a878304525f4aa5c5e7; no voters:
I20260808 06:03:54.897689 2903 raft_consensus.cc:2781] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 06:03:54.897840 2880 tablet.cc:2369] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 06:03:54.897944 2903 raft_consensus.cc:686] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 1 LEADER]: Becoming Leader. State: Replica: c4980ac04fca4a878304525f4aa5c5e7, State: Running, Role: LEADER
I20260808 06:03:54.898279 2903 consensus_queue.cc:237] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } }
I20260808 06:03:54.901497 2419 catalog_manager.cc:5495] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4980ac04fca4a878304525f4aa5c5e7 (127.13.229.65). New cstate: current_term: 1 leader_uuid: "c4980ac04fca4a878304525f4aa5c5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4980ac04fca4a878304525f4aa5c5e7" member_type: VOTER last_known_addr { host: "127.13.229.65" port: 36291 } health_report { overall_health: HEALTHY } } }
I20260808 06:03:54.924715 2893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.921471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.13.229.65:36291)
0808 06:03:54.921605 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:54.921609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:54.921621 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:03:54.921692 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:54.921694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:54.921747 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:54.921832 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:03:54.922048 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.922537 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.923335 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:54.923541 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:54.923617 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:54.923899 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:03:54.924337 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:03:54.924432 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:54.924496 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 06:03:54.924539 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:03:54.930483 2830 raft_consensus.cc:1261] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 1 FOLLOWER]: Refusing update from remote peer c4980ac04fca4a878304525f4aa5c5e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:54.930503 2698 raft_consensus.cc:1261] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 1 FOLLOWER]: Refusing update from remote peer c4980ac04fca4a878304525f4aa5c5e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:03:54.931237 2907 consensus_queue.cc:1035] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a219c0580e574282b77d86bf337aec6e" member_type: VOTER last_known_addr { host: "127.13.229.66" port: 40825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:54.931443 2903 consensus_queue.cc:1035] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3804dd02f1542899d2eef002ea1f444" member_type: VOTER last_known_addr { host: "127.13.229.67" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:03:54.948442 2419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:54.948634 2419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 06:03:54.951376 2419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c137366a59b4434bb138be31b411990: Sending DeleteTablet for 3 replicas of tablet 3f5bb7aef3634fdab75cbae422138a93
I20260808 06:03:54.952225 2810 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:54 UTC) from {username='slave'} at 127.0.0.1:55650
I20260808 06:03:54.952243 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:54 UTC) from {username='slave'} at 127.0.0.1:56020
I20260808 06:03:54.952724 2920 tablet_replica.cc:330] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: stopping tablet replica
I20260808 06:03:54.952940 2677 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f5bb7aef3634fdab75cbae422138a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 06:03:54 UTC) from {username='slave'} at 127.0.0.1:52704
I20260808 06:03:54.953158 2920 raft_consensus.cc:2229] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:54.953323 2921 tablet_replica.cc:330] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7: stopping tablet replica
I20260808 06:03:54.953397 2922 tablet_replica.cc:330] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: stopping tablet replica
I20260808 06:03:54.953585 2921 raft_consensus.cc:2229] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 06:03:54.953692 2920 raft_consensus.cc:2258] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:54.953742 2922 raft_consensus.cc:2229] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 06:03:54.953785 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2479
I20260808 06:03:54.954002 2921 raft_consensus.cc:2258] T 3f5bb7aef3634fdab75cbae422138a93 P c4980ac04fca4a878304525f4aa5c5e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:54.954941 2922 raft_consensus.cc:2258] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 06:03:54.956717 2922 ts_tablet_manager.cc:1905] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:54.957728 2920 ts_tablet_manager.cc:1905] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 06:03:54.959442 2922 ts_tablet_manager.cc:1918] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:54.959564 2922 log.cc:1196] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-1/wal/wals/3f5bb7aef3634fdab75cbae422138a93
I20260808 06:03:54.959933 2922 ts_tablet_manager.cc:1939] T 3f5bb7aef3634fdab75cbae422138a93 P a219c0580e574282b77d86bf337aec6e: Deleting consensus metadata
I20260808 06:03:54.961160 2404 catalog_manager.cc:4841] TS a219c0580e574282b77d86bf337aec6e (127.13.229.66:40825): tablet 3f5bb7aef3634fdab75cbae422138a93 (table test-table [id=ff9520eb4cac48649860a4be6f4d6d98]) successfully deleted
I20260808 06:03:54.961479 2920 ts_tablet_manager.cc:1918] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 06:03:54.961581 2920 log.cc:1196] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Deleting WAL directory at /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786168783448266-14229-0/minicluster-data/ts-2/wal/wals/3f5bb7aef3634fdab75cbae422138a93
I20260808 06:03:54.961932 2920 ts_tablet_manager.cc:1939] T 3f5bb7aef3634fdab75cbae422138a93 P a3804dd02f1542899d2eef002ea1f444: Deleting consensus metadata
I20260808 06:03:54.963239 2404 catalog_manager.cc:4841] TS a3804dd02f1542899d2eef002ea1f444 (127.13.229.67:44187): tablet 3f5bb7aef3634fdab75cbae422138a93 (table test-table [id=ff9520eb4cac48649860a4be6f4d6d98]) successfully deleted
W20260808 06:03:54.964825 2406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.229.65:36291 (error 108)
I20260808 06:03:54.964879 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2614
I20260808 06:03:54.966770 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:54.966497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.65:36291 (local address 127.0.0.1:56040)
0808 06:03:54.966625 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 06:03:54.966699 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.65:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 06:03:54.966923 2406 catalog_manager.cc:4584] TS c4980ac04fca4a878304525f4aa5c5e7 (127.13.229.65:36291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f5bb7aef3634fdab75cbae422138a93: Network error: Client connection negotiation failed: client connection to 127.13.229.65:36291: connect: Connection refused (error 111)
I20260808 06:03:54.972987 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2748
I20260808 06:03:54.981457 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2386
2026-08-08T06:03:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1161 ms)
[----------] 6 tests from EncryptionPolicyTest (10298 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:55 dist-test-slave-zkpd krb5kdc[2926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:55 dist-test-slave-zkpd krb5kdc[2926](info): set up 2 sockets
Aug 08 06:03:55 dist-test-slave-zkpd krb5kdc[2926](info): commencing operation
krb5kdc: starting...
W20260808 06:03:56.045653 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[2926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:03:56.170266 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:56.282421 2939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:56.282724 2939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:56.282828 2939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:56.286918 2939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:56.287010 2939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:56.287050 2939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:56.287110 2939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:56.287151 2939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:56.292348 2939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:56.293437 2939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:56.294487 2939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:56.303035 2944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:56.303061 2945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:56.303159 2947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:56.303357 2939 server_base.cc:1034] running on GCE node
I20260808 06:03:56.304044 2939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:56.305146 2939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:56.306344 2939 hybrid_clock.cc:648] HybridClock initialized: now 1786169036306311 us; error 46 us; skew 500 ppm
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[2926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:56.309166 2939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:03:56.309999 2939 webserver.cc:460] Webserver started at http://127.0.0.1:34839/ using document root <none> and password file <none>
I20260808 06:03:56.310544 2939 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:56.310634 2939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:56.310865 2939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:56.312933 2939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "ad901b3c4f164ae196b7044e1dee3cf3"
format_stamp: "Formatted at 2026-08-08 06:03:56 on dist-test-slave-zkpd"
server_key: "8424cb1796cf96a61aac363c3aea3e31"
server_key_iv: "fd460fa44e7cf1247131a97c52c1eeb9"
server_key_version: "encryptionkey@0"
I20260808 06:03:56.313501 2939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "ad901b3c4f164ae196b7044e1dee3cf3"
format_stamp: "Formatted at 2026-08-08 06:03:56 on dist-test-slave-zkpd"
server_key: "8424cb1796cf96a61aac363c3aea3e31"
server_key_iv: "fd460fa44e7cf1247131a97c52c1eeb9"
server_key_version: "encryptionkey@0"
I20260808 06:03:56.317083 2939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:56.319384 2955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:56.320590 2939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:03:56.320726 2939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "ad901b3c4f164ae196b7044e1dee3cf3"
format_stamp: "Formatted at 2026-08-08 06:03:56 on dist-test-slave-zkpd"
server_key: "8424cb1796cf96a61aac363c3aea3e31"
server_key_iv: "fd460fa44e7cf1247131a97c52c1eeb9"
server_key_version: "encryptionkey@0"
I20260808 06:03:56.320833 2939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:56.347271 2939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:56.350498 2939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:56.350703 2939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:56.357913 2939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36115
I20260808 06:03:56.357942 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36115 every 8 connection(s)
I20260808 06:03:56.358852 2939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:56.361549 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:56.366943 3008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3: Bootstrap starting.
I20260808 06:03:56.367018 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 2939
I20260808 06:03:56.367167 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:56.367479 14229 external_mini_cluster.cc:1428] Setting key ae0ee13dbce5bc8c30861c1610c0141b
I20260808 06:03:56.369746 3008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:56.370635 3008 log.cc:824] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:56.372848 3008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3: No bootstrap required, opened a new log
I20260808 06:03:56.376752 3008 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260808 06:03:56.376958 3008 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:56.377020 3008 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad901b3c4f164ae196b7044e1dee3cf3, State: Initialized, Role: FOLLOWER
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[2926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 06:03:56.377488 3008 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260808 06:03:56.377615 3008 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:56.377700 3008 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:56.377815 3008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:56.379025 3008 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260808 06:03:56.379390 3008 leader_election.cc:302] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad901b3c4f164ae196b7044e1dee3cf3; no voters:
I20260808 06:03:56.379747 3008 leader_election.cc:288] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:56.379843 3014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:56.380105 3014 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [term 1 LEADER]: Becoming Leader. State: Replica: ad901b3c4f164ae196b7044e1dee3cf3, State: Running, Role: LEADER
I20260808 06:03:56.380399 3014 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260808 06:03:56.381130 3008 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:56.381852 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260808 06:03:56.382025 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:56.382450 3023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:56.382409 3016 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad901b3c4f164ae196b7044e1dee3cf3. Latest consensus state: current_term: 1 leader_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad901b3c4f164ae196b7044e1dee3cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260808 06:03:56.383466 3016 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:56.384876 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:56.368916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51093 (local address 127.0.0.1:36115)
0808 06:03:56.369347 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:56.369366 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:56.369409 (+ 43us) server_negotiation.cc:407] Connection header received
0808 06:03:56.370290 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:56.370314 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:56.370745 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:56.371113 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:56.371997 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:56.373428 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:56.374282 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:56.374995 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:56.375148 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:56.377839 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:56.377845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:56.377863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:56.377901 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:56.379859 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:56.380524 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:56.380530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:56.380536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:56.380672 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:56.381085 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:56.381089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:56.381091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:56.382760 (+ 1669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:56.383563 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:56.383741 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 06:03:56.383992 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260808 06:03:56.385952 3023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:56.389578 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:03:56.392063 3023 catalog_manager.cc:1347] Generated new cluster ID: 75e4472321ff43a18b555e50dd9da33f
I20260808 06:03:56.392134 3023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:56.401793 3023 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:56.402911 3023 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:56.404918 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:56.391804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59125 (local address 127.0.0.1:36115)
0808 06:03:56.391931 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:56.391935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:56.396157 (+ 4222us) server_negotiation.cc:407] Connection header received
0808 06:03:56.396336 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:56.396339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:56.396386 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:56.396472 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:56.397196 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:56.398001 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:56.399082 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:56.399445 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:56.399548 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:56.401103 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:56.401107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:56.401110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:03:56.401136 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:56.402993 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:56.403517 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:56.403521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:56.403523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:56.403614 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:03:56.404013 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:03:56.404017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:03:56.404019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:03:56.404236 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:56.404376 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:56.404655 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 06:03:56.404750 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260808 06:03:56.414499 3023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad901b3c4f164ae196b7044e1dee3cf3: Generated new TSK 0
I20260808 06:03:56.415130 3023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:56.418352 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 2939
W20260808 06:03:56.426414 2924 connection.cc:535] client connection to 127.0.0.1:36115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36115): Connection reset by peer (error 104)
2026-08-08T06:03:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[3041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[3041](info): set up 2 sockets
Aug 08 06:03:56 dist-test-slave-zkpd krb5kdc[3041](info): commencing operation
krb5kdc: starting...
W20260808 06:03:57.475525 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:57 dist-test-slave-zkpd krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:03:57.603149 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:57.712249 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:57.712481 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:57.712520 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:57.716403 3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:57.716449 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:57.716467 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:57.716483 3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:57.716544 3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:57.721571 3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:57.722523 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:57.723500 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:57.731727 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:57.731739 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:57.731746 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:57.731983 3054 server_base.cc:1034] running on GCE node
I20260808 06:03:57.732663 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:57.733703 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:57.734917 3054 hybrid_clock.cc:648] HybridClock initialized: now 1786169037734900 us; error 36 us; skew 500 ppm
Aug 08 06:03:57 dist-test-slave-zkpd krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:57.737668 3054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:03:57.738477 3054 webserver.cc:460] Webserver started at http://127.0.0.1:41891/ using document root <none> and password file <none>
I20260808 06:03:57.738963 3054 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:57.739019 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:57.739173 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:57.741150 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "82006250f06e4693a4445f2779b79383"
format_stamp: "Formatted at 2026-08-08 06:03:57 on dist-test-slave-zkpd"
server_key: "0e9acba5e397f20d467f19a1211430dd"
server_key_iv: "3e3ffd32b6fcc23a902a2e01e6074f0d"
server_key_version: "encryptionkey@0"
I20260808 06:03:57.741668 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "82006250f06e4693a4445f2779b79383"
format_stamp: "Formatted at 2026-08-08 06:03:57 on dist-test-slave-zkpd"
server_key: "0e9acba5e397f20d467f19a1211430dd"
server_key_iv: "3e3ffd32b6fcc23a902a2e01e6074f0d"
server_key_version: "encryptionkey@0"
I20260808 06:03:57.745195 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:57.747438 3069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:57.748651 3054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:57.748780 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "82006250f06e4693a4445f2779b79383"
format_stamp: "Formatted at 2026-08-08 06:03:57 on dist-test-slave-zkpd"
server_key: "0e9acba5e397f20d467f19a1211430dd"
server_key_iv: "3e3ffd32b6fcc23a902a2e01e6074f0d"
server_key_version: "encryptionkey@0"
I20260808 06:03:57.748891 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:57.774436 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:57.777397 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:57.777629 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:57.784693 3054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38495
I20260808 06:03:57.784729 3121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38495 every 8 connection(s)
I20260808 06:03:57.785638 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:57.788362 3122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:57.789258 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3054
I20260808 06:03:57.789412 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:57.789747 14229 external_mini_cluster.cc:1428] Setting key 24b0e18fc9bdd8276c55338b0b3e1af7
I20260808 06:03:57.794499 3122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383: Bootstrap starting.
I20260808 06:03:57.796803 3122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:57.797541 3122 log.cc:824] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:57.798702 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:57.791140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50047 (local address 127.0.0.1:38495)
0808 06:03:57.791568 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:57.791582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:57.791618 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:03:57.792321 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:57.792340 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:57.792671 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:57.793008 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:57.793398 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:57.794361 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:57.795304 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:57.795765 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:57.795973 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:57.796901 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:57.796914 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:57.796936 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:57.797304 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:57.797357 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:57.797380 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:57.797526 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:57.797712 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 06:03:57.797845 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260808 06:03:57.799680 3122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383: No bootstrap required, opened a new log
I20260808 06:03:57.801613 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:03:57.803570 3122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } }
I20260808 06:03:57.803850 3122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:57.803960 3122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82006250f06e4693a4445f2779b79383, State: Initialized, Role: FOLLOWER
I20260808 06:03:57.804445 3122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } }
I20260808 06:03:57.804632 3122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:57.804700 3122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:57.804795 3122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:57.805783 3122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } }
I20260808 06:03:57.806137 3122 leader_election.cc:302] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82006250f06e4693a4445f2779b79383; no voters:
I20260808 06:03:57.806479 3122 leader_election.cc:288] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:57.806591 3132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:57.806816 3132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [term 1 LEADER]: Becoming Leader. State: Replica: 82006250f06e4693a4445f2779b79383, State: Running, Role: LEADER
I20260808 06:03:57.807148 3132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } }
I20260808 06:03:57.807797 3122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:57.808248 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:57.803602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58263 (local address 127.0.0.1:38495)
0808 06:03:57.803728 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:57.803732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:57.803962 (+ 230us) server_negotiation.cc:407] Connection header received
0808 06:03:57.804116 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:57.804119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:57.804161 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:57.804239 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:57.804520 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:57.805275 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:57.806143 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:03:57.806563 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:03:57.806683 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:03:57.807589 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:57.807592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:57.807595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:57.807607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:57.807634 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:57.807646 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:57.807736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:57.808071 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 06:03:57.808104 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:03:57.808426 3132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82006250f06e4693a4445f2779b79383. Latest consensus state: current_term: 1 leader_uuid: "82006250f06e4693a4445f2779b79383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } } }
I20260808 06:03:57.808545 3132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:57.808898 3134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82006250f06e4693a4445f2779b79383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82006250f06e4693a4445f2779b79383" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38495 } } }
I20260808 06:03:57.808992 3134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:57.809145 3136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:57.812192 3136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:57.817951 3136 catalog_manager.cc:1347] Generated new cluster ID: be4f660fcd044ad8b305f2c2655a49f5
I20260808 06:03:57.818024 3136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:57.844066 3136 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:57.845083 3136 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:57.850440 3136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82006250f06e4693a4445f2779b79383: Generated new TSK 0
I20260808 06:03:57.851086 3136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:57.858222 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3054
W20260808 06:03:57.865988 3039 connection.cc:535] client connection to 127.0.0.1:38495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38495): Connection reset by peer (error 104)
2026-08-08T06:03:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:57 dist-test-slave-zkpd krb5kdc[3153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:57 dist-test-slave-zkpd krb5kdc[3153](info): set up 2 sockets
Aug 08 06:03:57 dist-test-slave-zkpd krb5kdc[3153](info): commencing operation
krb5kdc: starting...
W20260808 06:03:58.926556 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:03:59 dist-test-slave-zkpd krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:03:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:03:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:03:59.054239 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:03:59.166241 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:03:59.166507 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:03:59.166548 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:03:59.170724 3166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:03:59.170785 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:03:59.170805 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:03:59.170821 3166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:03:59.170862 3166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:03:59.176391 3166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:03:59.177686 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:03:59.178877 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:03:59.187315 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:59.187311 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:03:59.187310 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:03:59.187672 3166 server_base.cc:1034] running on GCE node
I20260808 06:03:59.188326 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:03:59.189361 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:03:59.190549 3166 hybrid_clock.cc:648] HybridClock initialized: now 1786169039190527 us; error 33 us; skew 500 ppm
Aug 08 06:03:59 dist-test-slave-zkpd krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:03:59.193356 3166 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:03:59.194146 3166 webserver.cc:460] Webserver started at http://127.0.0.1:34971/ using document root <none> and password file <none>
I20260808 06:03:59.194638 3166 fs_manager.cc:359] Metadata directory not provided
I20260808 06:03:59.194693 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:03:59.194850 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:03:59.196826 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "e64743d863c44d969146d886b2b1f337"
format_stamp: "Formatted at 2026-08-08 06:03:59 on dist-test-slave-zkpd"
server_key: "0114fb10eba76ef5f6058bd782501657"
server_key_iv: "54c9305591ea48b5ef29fcd344b4eb6d"
server_key_version: "encryptionkey@0"
I20260808 06:03:59.197355 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "e64743d863c44d969146d886b2b1f337"
format_stamp: "Formatted at 2026-08-08 06:03:59 on dist-test-slave-zkpd"
server_key: "0114fb10eba76ef5f6058bd782501657"
server_key_iv: "54c9305591ea48b5ef29fcd344b4eb6d"
server_key_version: "encryptionkey@0"
I20260808 06:03:59.200891 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:03:59.203146 3182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:03:59.204576 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:03:59.204705 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "e64743d863c44d969146d886b2b1f337"
format_stamp: "Formatted at 2026-08-08 06:03:59 on dist-test-slave-zkpd"
server_key: "0114fb10eba76ef5f6058bd782501657"
server_key_iv: "54c9305591ea48b5ef29fcd344b4eb6d"
server_key_version: "encryptionkey@0"
I20260808 06:03:59.204805 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:03:59.209266 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:03:59.212430 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:03:59.212719 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:03:59.219499 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45417
I20260808 06:03:59.219581 3234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45417 every 8 connection(s)
I20260808 06:03:59.220471 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:03:59.223143 3235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:03:59.228475 3235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337: Bootstrap starting.
I20260808 06:03:59.230167 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3166
I20260808 06:03:59.230319 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:03:59.230607 14229 external_mini_cluster.cc:1428] Setting key 2b3ed13ac18d44dfdc2fa1fda87a3c7d
I20260808 06:03:59.230975 3235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337: Neither blocks nor log segments found. Creating new log.
I20260808 06:03:59.231880 3235 log.cc:824] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337: Log is configured to *not* fsync() on all Append() calls
I20260808 06:03:59.233803 3235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337: No bootstrap required, opened a new log
I20260808 06:03:59.235447 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:59.231694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39335 (local address 127.0.0.1:45417)
0808 06:03:59.232211 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:59.232225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:59.232260 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:03:59.232884 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:59.232900 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:59.233160 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:59.233499 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:59.234290 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:59.234295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:59.234307 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:59.234318 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:59.234368 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:59.234400 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:59.234518 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:59.234630 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 06:03:59.234786 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":169,"threads_started":1}
I20260808 06:03:59.237845 3235 raft_consensus.cc:348] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260808 06:03:59.238129 3235 raft_consensus.cc:374] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:03:59.238193 3235 raft_consensus.cc:729] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64743d863c44d969146d886b2b1f337, State: Initialized, Role: FOLLOWER
I20260808 06:03:59.238575 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:03:59.238621 3235 consensus_queue.cc:260] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260808 06:03:59.238791 3235 raft_consensus.cc:388] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:03:59.238878 3235 raft_consensus.cc:482] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:03:59.238996 3235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:03:59.239883 3235 raft_consensus.cc:504] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260808 06:03:59.240263 3235 leader_election.cc:302] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e64743d863c44d969146d886b2b1f337; no voters:
I20260808 06:03:59.240585 3235 leader_election.cc:288] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:03:59.240639 3244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:03:59.240849 3244 raft_consensus.cc:686] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [term 1 LEADER]: Becoming Leader. State: Replica: e64743d863c44d969146d886b2b1f337, State: Running, Role: LEADER
I20260808 06:03:59.241144 3244 consensus_queue.cc:237] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260808 06:03:59.242022 3235 sys_catalog.cc:564] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:03:59.242923 3245 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e64743d863c44d969146d886b2b1f337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260808 06:03:59.243073 3245 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:59.243351 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e64743d863c44d969146d886b2b1f337. Latest consensus state: current_term: 1 leader_uuid: "e64743d863c44d969146d886b2b1f337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e64743d863c44d969146d886b2b1f337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260808 06:03:59.243451 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337 [sys.catalog]: This master's current role is: LEADER
I20260808 06:03:59.243752 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:03:59.240982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57119 (local address 127.0.0.1:45417)
0808 06:03:59.241087 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:03:59.241091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:03:59.241716 (+ 625us) server_negotiation.cc:407] Connection header received
0808 06:03:59.241882 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:03:59.241885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:03:59.241920 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:03:59.242018 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:03:59.243025 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:03:59.243029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:03:59.243031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:03:59.243045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:03:59.243076 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:03:59.243093 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:03:59.243232 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 06:03:59.243561 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 06:03:59.243599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:03:59.243868 3254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:03:59.247921 3254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:03:59.253454 3254 catalog_manager.cc:1347] Generated new cluster ID: 5f9a4b8b1b2343cc831c2952abb61f58
I20260808 06:03:59.253525 3254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:03:59.268419 3254 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:03:59.268777 3254 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:03:59.273041 3254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e64743d863c44d969146d886b2b1f337: Generated new TSK 0
I20260808 06:03:59.273746 3254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:03:59.277303 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3166
2026-08-08T06:03:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:03:59 dist-test-slave-zkpd krb5kdc[3266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:03:59 dist-test-slave-zkpd krb5kdc[3266](info): set up 2 sockets
Aug 08 06:03:59 dist-test-slave-zkpd krb5kdc[3266](info): commencing operation
krb5kdc: starting...
W20260808 06:04:00.329114 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:00.455842 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:00.570649 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:00.570919 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:00.570961 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:00.574942 3279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:00.574998 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:00.575018 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:00.575034 3279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:00.575049 3279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:00.583709 3279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:00.585129 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:00.586176 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:00.594691 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:00.594687 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:00.594748 3279 server_base.cc:1034] running on GCE node
W20260808 06:04:00.594846 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:00.595562 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:00.596626 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:00.597784 3279 hybrid_clock.cc:648] HybridClock initialized: now 1786169040597759 us; error 36 us; skew 500 ppm
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:00.603866 3279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:00.604736 3279 webserver.cc:460] Webserver started at http://127.0.0.1:38749/ using document root <none> and password file <none>
I20260808 06:04:00.605280 3279 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:00.605341 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:00.605576 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:00.607587 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "fc19406567fe4ce2b9b6092f10cae5e6"
format_stamp: "Formatted at 2026-08-08 06:04:00 on dist-test-slave-zkpd"
server_key: "7ba57e68b8d39efc34aee059b6e6a908"
server_key_iv: "5672892c9a07d55c04588f203d0ef640"
server_key_version: "encryptionkey@0"
I20260808 06:04:00.608188 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "fc19406567fe4ce2b9b6092f10cae5e6"
format_stamp: "Formatted at 2026-08-08 06:04:00 on dist-test-slave-zkpd"
server_key: "7ba57e68b8d39efc34aee059b6e6a908"
server_key_iv: "5672892c9a07d55c04588f203d0ef640"
server_key_version: "encryptionkey@0"
I20260808 06:04:00.611811 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260808 06:04:00.614327 3295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:00.615550 3279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:04:00.615688 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "fc19406567fe4ce2b9b6092f10cae5e6"
format_stamp: "Formatted at 2026-08-08 06:04:00 on dist-test-slave-zkpd"
server_key: "7ba57e68b8d39efc34aee059b6e6a908"
server_key_iv: "5672892c9a07d55c04588f203d0ef640"
server_key_version: "encryptionkey@0"
I20260808 06:04:00.615796 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:00.634253 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:00.637125 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:00.637334 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:00.644464 3279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41185
I20260808 06:04:00.644481 3348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41185 every 8 connection(s)
I20260808 06:04:00.645457 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:00.648234 3349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:00.652997 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3279
I20260808 06:04:00.653146 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:00.653435 14229 external_mini_cluster.cc:1428] Setting key 518f544292f9b4d61e84ca739ccc8322
I20260808 06:04:00.653937 3349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: Bootstrap starting.
I20260808 06:04:00.656613 3349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:00.657531 3349 log.cc:824] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:00.659767 3349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: No bootstrap required, opened a new log
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 06:04:00.664317 3349 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } }
I20260808 06:04:00.664640 3349 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:00.664713 3349 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc19406567fe4ce2b9b6092f10cae5e6, State: Initialized, Role: FOLLOWER
I20260808 06:04:00.665246 3349 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } }
I20260808 06:04:00.665453 3349 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:00.665536 3349 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:00.665650 3349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:00.666797 3349 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } }
I20260808 06:04:00.667182 3349 leader_election.cc:302] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc19406567fe4ce2b9b6092f10cae5e6; no voters:
I20260808 06:04:00.667555 3349 leader_election.cc:288] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:00.667685 3354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:00.667949 3354 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [term 1 LEADER]: Becoming Leader. State: Replica: fc19406567fe4ce2b9b6092f10cae5e6, State: Running, Role: LEADER
I20260808 06:04:00.668284 3354 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } }
I20260808 06:04:00.668788 3349 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:00.669739 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:00.654910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35299 (local address 127.0.0.1:41185)
0808 06:04:00.655331 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:00.655351 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:00.655394 (+ 43us) server_negotiation.cc:407] Connection header received
0808 06:04:00.656237 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:00.656261 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:00.656702 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:00.657092 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:00.657884 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:00.659302 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:00.660310 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:00.660992 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:00.661162 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:00.664161 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:00.664167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:00.664182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:00.664214 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:00.666345 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:00.666770 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:00.666779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:00.666785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:00.666965 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:00.667265 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:00.667269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:00.667270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:00.667795 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:00.668029 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:00.668439 (+ 410us) server_negotiation.cc:299] Negotiation successful
0808 06:04:00.668738 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260808 06:04:00.669905 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } } }
I20260808 06:04:00.670010 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:00.670258 3355 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc19406567fe4ce2b9b6092f10cae5e6. Latest consensus state: current_term: 1 leader_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc19406567fe4ce2b9b6092f10cae5e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41185 } } }
I20260808 06:04:00.670356 3355 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:00.673496 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 06:04:00.674396 3369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:00.674494 3369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:00.674597 3363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:00.675446 3363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:00.681182 3363 catalog_manager.cc:1347] Generated new cluster ID: 149da779e74348c9ac94f9339c267ffe
I20260808 06:04:00.681242 3363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:00.685092 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:00.675743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44199 (local address 127.0.0.1:41185)
0808 06:04:00.675848 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:00.675852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:00.677291 (+ 1439us) server_negotiation.cc:407] Connection header received
0808 06:04:00.677464 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:00.677468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:00.677514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:00.677603 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:00.678264 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:00.678757 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:00.679777 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:00.680152 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:00.680263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:00.681864 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:00.681868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:00.681872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:00.681902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:00.683353 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:00.683806 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:00.683812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:00.683814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:00.683938 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:00.684229 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:00.684233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:00.684234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:00.684410 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:00.684545 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:00.684781 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 06:04:00.684896 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 06:04:00.699797 3363 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:00.700730 3363 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:00.709932 3363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc19406567fe4ce2b9b6092f10cae5e6: Generated new TSK 0
I20260808 06:04:00.710726 3363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:00.720286 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3279
2026-08-08T06:04:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3380](info): set up 2 sockets
Aug 08 06:04:00 dist-test-slave-zkpd krb5kdc[3380](info): commencing operation
krb5kdc: starting...
W20260808 06:04:01.777406 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:01 dist-test-slave-zkpd krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:01.907189 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:02.023478 3393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:02.023836 3393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:02.023958 3393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:02.028038 3393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:02.028152 3393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:02.028190 3393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:02.028228 3393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:02.028264 3393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:02.033761 3393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:02.034955 3393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:02.036093 3393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:02.044898 3398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:02.044906 3401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:02.044920 3399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:02.044912 3393 server_base.cc:1034] running on GCE node
I20260808 06:04:02.045753 3393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:02.046921 3393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:02.048122 3393 hybrid_clock.cc:648] HybridClock initialized: now 1786169042048106 us; error 25 us; skew 500 ppm
Aug 08 06:04:02 dist-test-slave-zkpd krb5kdc[3380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:02.050865 3393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:02.051772 3393 webserver.cc:460] Webserver started at http://127.0.0.1:46081/ using document root <none> and password file <none>
I20260808 06:04:02.052392 3393 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:02.052486 3393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:02.052727 3393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:02.054817 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "c9c9c51814f540a0991af423038c5367"
format_stamp: "Formatted at 2026-08-08 06:04:02 on dist-test-slave-zkpd"
server_key: "e988d36ff026e672949c131ceed81a5a"
server_key_iv: "2946de24e961be1068c842fe4367626e"
server_key_version: "encryptionkey@0"
I20260808 06:04:02.055413 3393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "c9c9c51814f540a0991af423038c5367"
format_stamp: "Formatted at 2026-08-08 06:04:02 on dist-test-slave-zkpd"
server_key: "e988d36ff026e672949c131ceed81a5a"
server_key_iv: "2946de24e961be1068c842fe4367626e"
server_key_version: "encryptionkey@0"
I20260808 06:04:02.059279 3393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 06:04:02.061755 3408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:02.063014 3393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:04:02.063158 3393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "c9c9c51814f540a0991af423038c5367"
format_stamp: "Formatted at 2026-08-08 06:04:02 on dist-test-slave-zkpd"
server_key: "e988d36ff026e672949c131ceed81a5a"
server_key_iv: "2946de24e961be1068c842fe4367626e"
server_key_version: "encryptionkey@0"
I20260808 06:04:02.063270 3393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:02.084998 3393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:02.087909 3393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:02.088127 3393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:02.095667 3393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38601
I20260808 06:04:02.095693 3461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38601 every 8 connection(s)
I20260808 06:04:02.096699 3393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:02.099515 3462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:02.105018 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3393
I20260808 06:04:02.105176 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:02.105346 3462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367: Bootstrap starting.
I20260808 06:04:02.105516 14229 external_mini_cluster.cc:1428] Setting key c3a2f945da0ccc58beb63936c4f23070
I20260808 06:04:02.107995 3462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:02.108803 3462 log.cc:824] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:02.110960 3462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367: No bootstrap required, opened a new log
I20260808 06:04:02.114612 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:02.106860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48613 (local address 127.0.0.1:38601)
0808 06:04:02.107283 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:02.107300 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:02.107337 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:04:02.108045 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:02.108067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:02.108464 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:02.108747 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:02.109139 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:02.110083 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:02.111262 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:02.111724 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:02.111859 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:02.112861 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:02.112866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:02.112880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:02.113264 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:02.113323 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:02.113346 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:02.113431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:02.113611 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 06:04:02.113881 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260808 06:04:02.115173 3462 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260808 06:04:02.115392 3462 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:02.115476 3462 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c9c51814f540a0991af423038c5367, State: Initialized, Role: FOLLOWER
I20260808 06:04:02.116021 3462 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260808 06:04:02.116178 3462 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:02.116245 3462 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:02.116343 3462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:02.117432 3462 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260808 06:04:02.117678 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:02.117807 3462 leader_election.cc:302] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9c9c51814f540a0991af423038c5367; no voters:
I20260808 06:04:02.118126 3462 leader_election.cc:288] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:02.118228 3467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:02.118461 3467 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [term 1 LEADER]: Becoming Leader. State: Replica: c9c9c51814f540a0991af423038c5367, State: Running, Role: LEADER
I20260808 06:04:02.118906 3467 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260808 06:04:02.119513 3462 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:02.120873 3469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c9c51814f540a0991af423038c5367. Latest consensus state: current_term: 1 leader_uuid: "c9c9c51814f540a0991af423038c5367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260808 06:04:02.121001 3469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:02.121369 3468 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9c9c51814f540a0991af423038c5367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c9c51814f540a0991af423038c5367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260808 06:04:02.121461 3483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:02.121482 3468 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:02.124708 3483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:02.127720 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:02.120362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38319 (local address 127.0.0.1:38601)
0808 06:04:02.120500 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:02.120505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:02.123304 (+ 2799us) server_negotiation.cc:407] Connection header received
0808 06:04:02.123493 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:02.123496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:02.123541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:02.123591 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:02.124010 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:02.124773 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:02.125877 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:02.126210 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:02.126321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:02.127226 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:02.127230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:02.127233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:02.127247 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:02.127277 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:02.127291 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:02.127371 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:02.127519 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 06:04:02.127576 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 06:04:02.130656 3483 catalog_manager.cc:1347] Generated new cluster ID: 95ad5874a5be42b3b29fa4104f5f49c3
I20260808 06:04:02.130716 3483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:02.152405 3483 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:02.153700 3483 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:02.172429 3483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c9c51814f540a0991af423038c5367: Generated new TSK 0
I20260808 06:04:02.173311 3483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:02.178370 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3393
2026-08-08T06:04:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:02 dist-test-slave-zkpd krb5kdc[3493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:02 dist-test-slave-zkpd krb5kdc[3493](info): set up 2 sockets
Aug 08 06:04:02 dist-test-slave-zkpd krb5kdc[3493](info): commencing operation
krb5kdc: starting...
W20260808 06:04:03.249359 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:03 dist-test-slave-zkpd krb5kdc[3493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:03.374610 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:03.504905 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:03.505162 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:03.505201 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:03.509213 3506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:03.509265 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:03.509285 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:03.509302 3506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:03.509317 3506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:03.514492 3506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:03.515480 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:03.516507 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:03.525063 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:03.525233 3506 server_base.cc:1034] running on GCE node
W20260808 06:04:03.525086 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:03.525076 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:03.525902 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:03.526960 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:03.528146 3506 hybrid_clock.cc:648] HybridClock initialized: now 1786169043528119 us; error 37 us; skew 500 ppm
Aug 08 06:04:03 dist-test-slave-zkpd krb5kdc[3493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:03.530761 3506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:03.531667 3506 webserver.cc:460] Webserver started at http://127.0.0.1:36867/ using document root <none> and password file <none>
I20260808 06:04:03.532285 3506 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:03.532378 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:03.532610 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:03.534626 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "7313e95b96fc4b58a738f3486e24ab99"
format_stamp: "Formatted at 2026-08-08 06:04:03 on dist-test-slave-zkpd"
server_key: "7793766ee23966d5925c937a2cfe69ce"
server_key_iv: "9a3e619a3d9231dd74358af7a2de58ee"
server_key_version: "encryptionkey@0"
I20260808 06:04:03.535205 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "7313e95b96fc4b58a738f3486e24ab99"
format_stamp: "Formatted at 2026-08-08 06:04:03 on dist-test-slave-zkpd"
server_key: "7793766ee23966d5925c937a2cfe69ce"
server_key_iv: "9a3e619a3d9231dd74358af7a2de58ee"
server_key_version: "encryptionkey@0"
I20260808 06:04:03.538898 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 06:04:03.541350 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:03.542562 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:03.542702 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "7313e95b96fc4b58a738f3486e24ab99"
format_stamp: "Formatted at 2026-08-08 06:04:03 on dist-test-slave-zkpd"
server_key: "7793766ee23966d5925c937a2cfe69ce"
server_key_iv: "9a3e619a3d9231dd74358af7a2de58ee"
server_key_version: "encryptionkey@0"
I20260808 06:04:03.542811 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:03.547291 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:03.550086 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:03.550287 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:03.557477 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40821
I20260808 06:04:03.557511 3574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40821 every 8 connection(s)
I20260808 06:04:03.558374 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:03.561192 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3506
I20260808 06:04:03.561367 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:03.561538 3575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:03.561722 14229 external_mini_cluster.cc:1428] Setting key 5db95c44c8134cffb876b95006d443e4
I20260808 06:04:03.567567 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:03.563301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51155 (local address 127.0.0.1:40821)
0808 06:04:03.563768 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:03.563788 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:03.563834 (+ 46us) server_negotiation.cc:407] Connection header received
0808 06:04:03.564592 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:03.564617 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:03.565015 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:03.565418 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:03.566120 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:03.566126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:03.566141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:03.566153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:03.566213 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:03.566239 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:03.566384 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:03.566454 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 06:04:03.566618 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260808 06:04:03.569193 3575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99: Bootstrap starting.
I20260808 06:04:03.570349 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:03.571800 3575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:03.572697 3575 log.cc:824] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:03.574347 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:03.572473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43805 (local address 127.0.0.1:40821)
0808 06:04:03.572603 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:03.572609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:03.572862 (+ 253us) server_negotiation.cc:407] Connection header received
0808 06:04:03.572950 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:03.572952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:03.572980 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:03.573044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:03.573953 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:03.573956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:03.573958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:03.573968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:03.573991 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:03.574001 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:03.574074 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:03.574192 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 06:04:03.574221 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 06:04:03.575109 3575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99: No bootstrap required, opened a new log
I20260808 06:04:03.578826 3575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260808 06:04:03.579082 3575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:03.579178 3575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7313e95b96fc4b58a738f3486e24ab99, State: Initialized, Role: FOLLOWER
I20260808 06:04:03.579628 3575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260808 06:04:03.579795 3575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:03.579887 3575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:03.580052 3575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:03.581009 3575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260808 06:04:03.581375 3575 leader_election.cc:302] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7313e95b96fc4b58a738f3486e24ab99; no voters:
I20260808 06:04:03.581692 3575 leader_election.cc:288] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:03.581804 3585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:03.582077 3585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [term 1 LEADER]: Becoming Leader. State: Replica: 7313e95b96fc4b58a738f3486e24ab99, State: Running, Role: LEADER
I20260808 06:04:03.582441 3585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260808 06:04:03.583024 3575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:03.584184 3586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7313e95b96fc4b58a738f3486e24ab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20260808 06:04:03.584331 3586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:03.584642 3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7313e95b96fc4b58a738f3486e24ab99. Latest consensus state: current_term: 1 leader_uuid: "7313e95b96fc4b58a738f3486e24ab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7313e95b96fc4b58a738f3486e24ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20260808 06:04:03.584740 3594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:03.584803 3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:03.587697 3594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:03.593116 3594 catalog_manager.cc:1347] Generated new cluster ID: fe8630b422994326a903175916f75fa4
I20260808 06:04:03.593182 3594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:03.600539 3594 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:03.600845 3594 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:03.610324 3594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7313e95b96fc4b58a738f3486e24ab99: Generated new TSK 0
I20260808 06:04:03.610934 3594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:03.616473 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3506
2026-08-08T06:04:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:03 dist-test-slave-zkpd krb5kdc[3607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:03 dist-test-slave-zkpd krb5kdc[3607](info): set up 2 sockets
Aug 08 06:04:03 dist-test-slave-zkpd krb5kdc[3607](info): commencing operation
krb5kdc: starting...
W20260808 06:04:04.674628 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:04 dist-test-slave-zkpd krb5kdc[3607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:04.802853 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:04.913785 3620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:04.914033 3620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:04.914073 3620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:04.918073 3620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:04.918126 3620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:04.918145 3620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:04.918162 3620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:04.918223 3620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:04.923341 3620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:04.924362 3620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:04.925372 3620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:04.933679 3625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:04.933911 3620 server_base.cc:1034] running on GCE node
W20260808 06:04:04.933719 3626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:04.933745 3628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:04.934566 3620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:04.935602 3620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:04.936779 3620 hybrid_clock.cc:648] HybridClock initialized: now 1786169044936738 us; error 50 us; skew 500 ppm
Aug 08 06:04:04 dist-test-slave-zkpd krb5kdc[3607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:04.939560 3620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:04.940496 3620 webserver.cc:460] Webserver started at http://127.0.0.1:37545/ using document root <none> and password file <none>
I20260808 06:04:04.941000 3620 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:04.941057 3620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:04.941216 3620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:04.943167 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "ff17f6b001234fedb8c8d4c3d9931cdd"
format_stamp: "Formatted at 2026-08-08 06:04:04 on dist-test-slave-zkpd"
server_key: "0cf6f1e3059b3a9207d4f0b12f6f5ce2"
server_key_iv: "e15916f8de44ef70ec39ac7a960b566b"
server_key_version: "encryptionkey@0"
I20260808 06:04:04.943671 3620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "ff17f6b001234fedb8c8d4c3d9931cdd"
format_stamp: "Formatted at 2026-08-08 06:04:04 on dist-test-slave-zkpd"
server_key: "0cf6f1e3059b3a9207d4f0b12f6f5ce2"
server_key_iv: "e15916f8de44ef70ec39ac7a960b566b"
server_key_version: "encryptionkey@0"
I20260808 06:04:04.947216 3620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 06:04:04.949777 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:04.951028 3620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:04.951153 3620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "ff17f6b001234fedb8c8d4c3d9931cdd"
format_stamp: "Formatted at 2026-08-08 06:04:04 on dist-test-slave-zkpd"
server_key: "0cf6f1e3059b3a9207d4f0b12f6f5ce2"
server_key_iv: "e15916f8de44ef70ec39ac7a960b566b"
server_key_version: "encryptionkey@0"
I20260808 06:04:04.951259 3620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:04.966569 3620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:04.969676 3620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:04.969892 3620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:04.977162 3620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39443
I20260808 06:04:04.977191 3690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39443 every 8 connection(s)
I20260808 06:04:04.978106 3620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:04.978909 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3620
I20260808 06:04:04.979065 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:04.979390 14229 external_mini_cluster.cc:1428] Setting key 26dcdbc92fb110b82dfeda9b054576c8
I20260808 06:04:04.981616 3691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:04.987273 3691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd: Bootstrap starting.
Aug 08 06:04:04 dist-test-slave-zkpd krb5kdc[3607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 06:04:04.989657 3691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:04.990460 3691 log.cc:824] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:04.992784 3691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd: No bootstrap required, opened a new log
I20260808 06:04:04.993652 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:04.980915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:45479 (local address 127.0.0.1:39443)
0808 06:04:04.981413 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:04.981428 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:04.981465 (+ 37us) server_negotiation.cc:407] Connection header received
0808 06:04:04.982148 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:04.982166 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:04.982510 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:04.982850 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:04.983683 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:04.984737 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:04.985619 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:04.986084 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:04.986257 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:04.988886 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:04.988891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:04.988904 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:04.988929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:04.990641 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:04.991244 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:04.991253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:04.991259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:04.991375 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:04.991723 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:04.991725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:04.991726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:04.992163 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:04.992327 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:04.992683 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 06:04:04.992944 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":175,"threads_started":1}
I20260808 06:04:04.996829 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:04.997059 3691 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260808 06:04:04.997323 3691 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:04.997388 3691 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff17f6b001234fedb8c8d4c3d9931cdd, State: Initialized, Role: FOLLOWER
I20260808 06:04:04.997846 3691 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260808 06:04:04.998008 3691 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:04.998095 3691 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:04.998242 3691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:04.999325 3691 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260808 06:04:04.999655 3691 leader_election.cc:302] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff17f6b001234fedb8c8d4c3d9931cdd; no voters:
I20260808 06:04:05.000005 3691 leader_election.cc:288] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:05.000123 3702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:05.000352 3702 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [term 1 LEADER]: Becoming Leader. State: Replica: ff17f6b001234fedb8c8d4c3d9931cdd, State: Running, Role: LEADER
I20260808 06:04:05.000746 3702 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260808 06:04:05.001304 3691 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:05.002461 3704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff17f6b001234fedb8c8d4c3d9931cdd. Latest consensus state: current_term: 1 leader_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } } }
I20260808 06:04:05.002599 3704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:05.002877 3703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff17f6b001234fedb8c8d4c3d9931cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } } }
I20260808 06:04:05.002982 3703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:05.003041 3711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:05.006467 3711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:05.008411 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:04.999123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:53025 (local address 127.0.0.1:39443)
0808 06:04:04.999247 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:04.999253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:04.999463 (+ 210us) server_negotiation.cc:407] Connection header received
0808 06:04:04.999598 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:04.999601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:04.999641 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:04.999727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:05.000570 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:05.001358 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:05.002571 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:05.002952 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:05.003059 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:05.004793 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:05.004797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:05.004799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:05.004823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:05.006510 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:05.007028 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:05.007031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:05.007032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:05.007117 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:05.007516 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:05.007518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:05.007519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:05.007674 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:05.007780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:05.008158 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 06:04:05.008255 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:04:05.011904 3711 catalog_manager.cc:1347] Generated new cluster ID: d399a6e2a40545688f408876d4d3b5e1
I20260808 06:04:05.011986 3711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:05.025000 3711 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:05.026108 3711 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:05.040318 3711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff17f6b001234fedb8c8d4c3d9931cdd: Generated new TSK 0
I20260808 06:04:05.041095 3711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:05.049714 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3620
W20260808 06:04:05.057971 3605 connection.cc:535] client connection to 127.0.0.1:39443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39443): Connection reset by peer (error 104)
2026-08-08T06:04:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:05 dist-test-slave-zkpd krb5kdc[3723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:05 dist-test-slave-zkpd krb5kdc[3723](info): set up 2 sockets
Aug 08 06:04:05 dist-test-slave-zkpd krb5kdc[3723](info): commencing operation
krb5kdc: starting...
W20260808 06:04:06.121551 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:06 dist-test-slave-zkpd krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:06.251998 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:06.364298 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:06.364604 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:06.364708 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:06.368683 3738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:06.368772 3738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:06.368820 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:06.368860 3738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:06.368894 3738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:06.374130 3738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:06.375133 3738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:06.376217 3738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:06.384433 3744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:06.384445 3746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:06.384445 3743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:06.384608 3738 server_base.cc:1034] running on GCE node
I20260808 06:04:06.385324 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:06.386400 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:06.387566 3738 hybrid_clock.cc:648] HybridClock initialized: now 1786169046387546 us; error 29 us; skew 500 ppm
Aug 08 06:04:06 dist-test-slave-zkpd krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:06.394555 3738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:06.395478 3738 webserver.cc:460] Webserver started at http://127.0.0.1:32979/ using document root <none> and password file <none>
I20260808 06:04:06.396118 3738 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:06.396212 3738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:06.396452 3738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:06.398480 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "2990288ec66e4c05b3731432ca734250"
format_stamp: "Formatted at 2026-08-08 06:04:06 on dist-test-slave-zkpd"
server_key: "866dc98310222c04d8f8a1c9463e7632"
server_key_iv: "873f114183385be7bcee7ad5cdfa0e40"
server_key_version: "encryptionkey@0"
I20260808 06:04:06.399062 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "2990288ec66e4c05b3731432ca734250"
format_stamp: "Formatted at 2026-08-08 06:04:06 on dist-test-slave-zkpd"
server_key: "866dc98310222c04d8f8a1c9463e7632"
server_key_iv: "873f114183385be7bcee7ad5cdfa0e40"
server_key_version: "encryptionkey@0"
I20260808 06:04:06.402755 3738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 06:04:06.405256 3753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:06.406414 3738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:06.406560 3738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "2990288ec66e4c05b3731432ca734250"
format_stamp: "Formatted at 2026-08-08 06:04:06 on dist-test-slave-zkpd"
server_key: "866dc98310222c04d8f8a1c9463e7632"
server_key_iv: "873f114183385be7bcee7ad5cdfa0e40"
server_key_version: "encryptionkey@0"
I20260808 06:04:06.406669 3738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:06.430872 3738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:06.433861 3738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:06.434069 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:06.441668 3738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38085
I20260808 06:04:06.441756 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38085 every 8 connection(s)
I20260808 06:04:06.442646 3738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:06.445504 3807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:06.448563 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3738
I20260808 06:04:06.448732 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:06.449044 14229 external_mini_cluster.cc:1428] Setting key ac47e3a93a08062ef2d28be36c145c18
I20260808 06:04:06.451655 3807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250: Bootstrap starting.
I20260808 06:04:06.453897 3807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:06.454660 3807 log.cc:824] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:06.456696 3807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250: No bootstrap required, opened a new log
I20260808 06:04:06.458187 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:06.450491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58377 (local address 127.0.0.1:38085)
0808 06:04:06.450932 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:06.450946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:06.450982 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:04:06.451607 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:06.451623 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:06.452072 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:06.452405 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:06.452819 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:06.453813 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:06.454789 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:06.455265 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:06.455444 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:06.456442 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:06.456447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:06.456460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:06.456883 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:06.456940 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:06.456963 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:06.457108 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:06.457307 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 06:04:06.457446 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260808 06:04:06.460840 3807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260808 06:04:06.461110 3807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:06.461192 3807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2990288ec66e4c05b3731432ca734250, State: Initialized, Role: FOLLOWER
I20260808 06:04:06.461460 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:06.461694 3807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260808 06:04:06.461843 3807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:06.461905 3807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:06.462015 3807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:06.463194 3807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260808 06:04:06.463565 3807 leader_election.cc:302] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2990288ec66e4c05b3731432ca734250; no voters:
I20260808 06:04:06.463939 3807 leader_election.cc:288] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:06.464373 3819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:06.464655 3819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [term 1 LEADER]: Becoming Leader. State: Replica: 2990288ec66e4c05b3731432ca734250, State: Running, Role: LEADER
I20260808 06:04:06.465040 3819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260808 06:04:06.465221 3807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:06.466260 3819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2990288ec66e4c05b3731432ca734250. Latest consensus state: current_term: 1 leader_uuid: "2990288ec66e4c05b3731432ca734250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260808 06:04:06.466431 3819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:06.466676 3820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2990288ec66e4c05b3731432ca734250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2990288ec66e4c05b3731432ca734250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260808 06:04:06.466859 3820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:06.466921 3827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:06.468894 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:06.463770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:56625 (local address 127.0.0.1:38085)
0808 06:04:06.463939 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:06.463943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:06.464216 (+ 273us) server_negotiation.cc:407] Connection header received
0808 06:04:06.464335 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:06.464338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:06.464382 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:06.464465 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:06.464837 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:06.465567 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:06.466789 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:06.467120 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:06.467194 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:06.468314 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:06.468321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:06.468324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:06.468339 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:06.468373 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:06.468389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:06.468470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:06.468728 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 06:04:06.468772 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 06:04:06.470409 3827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:06.476279 3827 catalog_manager.cc:1347] Generated new cluster ID: 338e03bdb22d45b3b6bedce0fdf54787
I20260808 06:04:06.476343 3827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:06.492595 3827 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:06.493842 3827 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:06.502357 3827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2990288ec66e4c05b3731432ca734250: Generated new TSK 0
I20260808 06:04:06.503062 3827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:06.514896 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3738
W20260808 06:04:06.522985 3721 connection.cc:535] client connection to 127.0.0.1:38085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38085): Connection reset by peer (error 104)
2026-08-08T06:04:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:06 dist-test-slave-zkpd krb5kdc[3840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:06 dist-test-slave-zkpd krb5kdc[3840](info): set up 2 sockets
Aug 08 06:04:06 dist-test-slave-zkpd krb5kdc[3840](info): commencing operation
krb5kdc: starting...
W20260808 06:04:07.576833 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:07 dist-test-slave-zkpd krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:07.702244 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:07.814688 3853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:07.814939 3853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:07.814980 3853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:07.818809 3853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:07.818857 3853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:07.818876 3853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:07.818892 3853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:07.818907 3853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:07.823837 3853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:07.824798 3853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:07.825706 3853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:07.834159 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:07.834152 3858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:07.834247 3853 server_base.cc:1034] running on GCE node
W20260808 06:04:07.834163 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:07.834995 3853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:07.836112 3853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:07.837299 3853 hybrid_clock.cc:648] HybridClock initialized: now 1786169047837281 us; error 28 us; skew 500 ppm
Aug 08 06:04:07 dist-test-slave-zkpd krb5kdc[3840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:07.840056 3853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:07.840840 3853 webserver.cc:460] Webserver started at http://127.0.0.1:37799/ using document root <none> and password file <none>
I20260808 06:04:07.841337 3853 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:07.841395 3853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:07.841549 3853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:07.843463 3853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "83e66da220bf4eb4a21d1da300a5036e"
format_stamp: "Formatted at 2026-08-08 06:04:07 on dist-test-slave-zkpd"
server_key: "0ebcefd1cc9ce3eca82e4e18117dd1dd"
server_key_iv: "c206b62dff5d5eaba65d9dec7edb13eb"
server_key_version: "encryptionkey@0"
I20260808 06:04:07.844070 3853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "83e66da220bf4eb4a21d1da300a5036e"
format_stamp: "Formatted at 2026-08-08 06:04:07 on dist-test-slave-zkpd"
server_key: "0ebcefd1cc9ce3eca82e4e18117dd1dd"
server_key_iv: "c206b62dff5d5eaba65d9dec7edb13eb"
server_key_version: "encryptionkey@0"
I20260808 06:04:07.847605 3853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 06:04:07.849887 3868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:07.851053 3853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:04:07.851209 3853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "83e66da220bf4eb4a21d1da300a5036e"
format_stamp: "Formatted at 2026-08-08 06:04:07 on dist-test-slave-zkpd"
server_key: "0ebcefd1cc9ce3eca82e4e18117dd1dd"
server_key_iv: "c206b62dff5d5eaba65d9dec7edb13eb"
server_key_version: "encryptionkey@0"
I20260808 06:04:07.851306 3853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:07.855762 3853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:07.858458 3853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:07.858651 3853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:07.865640 3853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45435
I20260808 06:04:07.865669 3921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45435 every 8 connection(s)
I20260808 06:04:07.866513 3853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:07.868754 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3853
I20260808 06:04:07.868916 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:07.869233 14229 external_mini_cluster.cc:1428] Setting key 2496c5fbe6b6c9c6820464323b57fbf7
I20260808 06:04:07.869396 3922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:07.874662 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:07.870599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:56817 (local address 127.0.0.1:45435)
0808 06:04:07.871030 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:07.871045 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:07.871078 (+ 33us) server_negotiation.cc:407] Connection header received
0808 06:04:07.872071 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:07.872091 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:07.872365 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:07.872695 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:07.873504 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:07.873508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:07.873520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:07.873530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:07.873582 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:07.873600 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:07.873769 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:07.873844 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 06:04:07.874007 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260808 06:04:07.876084 3922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e: Bootstrap starting.
I20260808 06:04:07.877560 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:07.878549 3922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:07.879334 3922 log.cc:824] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:07.881312 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:07.879972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:33141 (local address 127.0.0.1:45435)
0808 06:04:07.880083 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:07.880086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:07.880098 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:04:07.880128 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:07.880130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:07.880155 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:07.880219 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:07.880955 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:07.880959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:07.880962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:07.880974 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:07.881001 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:07.881013 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:07.881104 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:07.881166 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 06:04:07.881203 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:04:07.881815 3922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e: No bootstrap required, opened a new log
I20260808 06:04:07.885460 3922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 06:04:07.885716 3922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:07.885773 3922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e66da220bf4eb4a21d1da300a5036e, State: Initialized, Role: FOLLOWER
I20260808 06:04:07.886138 3922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 06:04:07.886247 3922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:07.886286 3922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:07.886353 3922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:07.887461 3922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 06:04:07.887756 3922 leader_election.cc:302] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83e66da220bf4eb4a21d1da300a5036e; no voters:
I20260808 06:04:07.888056 3922 leader_election.cc:288] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:07.888200 3932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:07.888463 3932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [term 1 LEADER]: Becoming Leader. State: Replica: 83e66da220bf4eb4a21d1da300a5036e, State: Running, Role: LEADER
I20260808 06:04:07.888762 3932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 06:04:07.889242 3922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:07.890385 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e66da220bf4eb4a21d1da300a5036e. Latest consensus state: current_term: 1 leader_uuid: "83e66da220bf4eb4a21d1da300a5036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260808 06:04:07.890523 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:07.890920 3941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:07.890865 3933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83e66da220bf4eb4a21d1da300a5036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83e66da220bf4eb4a21d1da300a5036e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260808 06:04:07.891036 3933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:07.893919 3941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:07.898945 3941 catalog_manager.cc:1347] Generated new cluster ID: 1539d9a7b1d64597a3c75dae9403044b
I20260808 06:04:07.899010 3941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:07.904948 3941 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:07.905232 3941 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:07.909184 3941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83e66da220bf4eb4a21d1da300a5036e: Generated new TSK 0
I20260808 06:04:07.909840 3941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:07.914767 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3853
2026-08-08T06:04:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:07 dist-test-slave-zkpd krb5kdc[3954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:07 dist-test-slave-zkpd krb5kdc[3954](info): set up 2 sockets
Aug 08 06:04:07 dist-test-slave-zkpd krb5kdc[3954](info): commencing operation
krb5kdc: starting...
W20260808 06:04:08.989573 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:09.113168 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:09.225168 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:09.225430 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:09.225473 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:09.229439 3967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:09.229493 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:09.229512 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:09.229529 3967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:09.229544 3967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:09.234797 3967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:09.235853 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:09.236904 3967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 06:04:09.245352 3967 server_base.cc:1034] running on GCE node
W20260808 06:04:09.245323 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:09.245304 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:09.245303 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:09.246117 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:09.247165 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:09.248350 3967 hybrid_clock.cc:648] HybridClock initialized: now 1786169049248337 us; error 24 us; skew 500 ppm
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[3954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:09.251065 3967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:09.251957 3967 webserver.cc:460] Webserver started at http://127.0.0.1:42369/ using document root <none> and password file <none>
I20260808 06:04:09.252506 3967 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:09.252599 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:09.252831 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:09.254845 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "4ea39cfeb6924deea93408e543a39297"
format_stamp: "Formatted at 2026-08-08 06:04:09 on dist-test-slave-zkpd"
server_key: "51f26abdff02903291f3f1b877080139"
server_key_iv: "592820697c58c098cc320766c69296d5"
server_key_version: "encryptionkey@0"
I20260808 06:04:09.255424 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "4ea39cfeb6924deea93408e543a39297"
format_stamp: "Formatted at 2026-08-08 06:04:09 on dist-test-slave-zkpd"
server_key: "51f26abdff02903291f3f1b877080139"
server_key_iv: "592820697c58c098cc320766c69296d5"
server_key_version: "encryptionkey@0"
I20260808 06:04:09.259092 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 06:04:09.261433 3982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:09.262555 3967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:09.262705 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "4ea39cfeb6924deea93408e543a39297"
format_stamp: "Formatted at 2026-08-08 06:04:09 on dist-test-slave-zkpd"
server_key: "51f26abdff02903291f3f1b877080139"
server_key_iv: "592820697c58c098cc320766c69296d5"
server_key_version: "encryptionkey@0"
I20260808 06:04:09.262812 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:09.273710 3967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:09.276410 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:09.276630 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:09.283521 3967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45839
I20260808 06:04:09.283595 4035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45839 every 8 connection(s)
I20260808 06:04:09.284497 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:09.287194 4036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:09.289286 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 3967
I20260808 06:04:09.289461 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:09.289743 14229 external_mini_cluster.cc:1428] Setting key 7bd84097d528ba18bbd9db925d222b13
I20260808 06:04:09.294284 4036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: Bootstrap starting.
I20260808 06:04:09.297029 4036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:09.298002 4036 log.cc:824] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[3954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 06:04:09.300302 4036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: No bootstrap required, opened a new log
I20260808 06:04:09.305052 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:09.291118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:36795 (local address 127.0.0.1:45839)
0808 06:04:09.291527 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:09.291543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:09.291579 (+ 36us) server_negotiation.cc:407] Connection header received
0808 06:04:09.292344 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:09.292365 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:09.292709 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:09.292995 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:09.293984 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:09.294995 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:09.296218 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:09.296678 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:09.296804 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:09.299692 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:09.299699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:09.299715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:09.299747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:09.301736 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:09.302245 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:09.302251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:09.302257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:09.302414 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:09.302751 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:09.302754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:09.302757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:09.303308 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:09.303530 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:09.303757 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 06:04:09.304039 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260808 06:04:09.305146 4036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 06:04:09.305414 4036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:09.305482 4036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea39cfeb6924deea93408e543a39297, State: Initialized, Role: FOLLOWER
I20260808 06:04:09.305960 4036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 06:04:09.306126 4036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:09.306200 4036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:09.306300 4036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:09.307399 4036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 06:04:09.307767 4036 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea39cfeb6924deea93408e543a39297; no voters:
I20260808 06:04:09.308192 4036 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:09.308279 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:09.308288 4041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:09.308502 4041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea39cfeb6924deea93408e543a39297, State: Running, Role: LEADER
I20260808 06:04:09.308843 4041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } }
I20260808 06:04:09.309542 4036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:09.310493 4043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea39cfeb6924deea93408e543a39297. Latest consensus state: current_term: 1 leader_uuid: "4ea39cfeb6924deea93408e543a39297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } } }
I20260808 06:04:09.310629 4043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:09.310935 4042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea39cfeb6924deea93408e543a39297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea39cfeb6924deea93408e543a39297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45839 } } }
I20260808 06:04:09.311038 4042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297 [sys.catalog]: This master's current role is: LEADER
W20260808 06:04:09.314509 4061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:09.314621 4061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:09.314720 4055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:09.315510 4055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:09.319631 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:09.310848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:45095 (local address 127.0.0.1:45839)
0808 06:04:09.310943 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:09.310946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:09.310961 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:04:09.311122 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:09.311125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:09.311178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:09.311251 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:09.312195 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:09.312934 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:09.314049 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:09.314379 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:09.314487 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:09.316292 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:09.316296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:09.316298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:09.316323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:09.317876 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:09.318353 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:09.318356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:09.318358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:09.318441 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:09.318772 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:09.318775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:09.318776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:09.318954 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:09.319107 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:09.319377 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 06:04:09.319469 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 06:04:09.321574 4055 catalog_manager.cc:1347] Generated new cluster ID: 32e19e5cd2524587bdde3405671d702e
I20260808 06:04:09.321635 4055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:09.330932 4055 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:09.332231 4055 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:09.340206 4055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea39cfeb6924deea93408e543a39297: Generated new TSK 0
I20260808 06:04:09.340831 4055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:09.347100 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 3967
W20260808 06:04:09.354655 3952 connection.cc:535] client connection to 127.0.0.1:45839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45839): Connection reset by peer (error 104)
2026-08-08T06:04:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[4067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[4067](info): set up 2 sockets
Aug 08 06:04:09 dist-test-slave-zkpd krb5kdc[4067](info): commencing operation
krb5kdc: starting...
W20260808 06:04:10.417344 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:10 dist-test-slave-zkpd krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:10.550917 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:10.665699 4080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:10.666004 4080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:10.666083 4080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:10.670110 4080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:10.670199 4080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:10.670238 4080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:10.670297 4080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:10.670341 4080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:10.675539 4080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:10.676640 4080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:10.677697 4080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:10.686435 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:10.686472 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:10.686391 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:10.686776 4080 server_base.cc:1034] running on GCE node
I20260808 06:04:10.687340 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:10.688494 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:10.689679 4080 hybrid_clock.cc:648] HybridClock initialized: now 1786169050689647 us; error 45 us; skew 500 ppm
Aug 08 06:04:10 dist-test-slave-zkpd krb5kdc[4067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:10.692498 4080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:10.693331 4080 webserver.cc:460] Webserver started at http://127.0.0.1:46753/ using document root <none> and password file <none>
I20260808 06:04:10.693893 4080 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:10.693989 4080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:10.694224 4080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:10.696312 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "42ac70082f2141ba94295b5345e45357"
format_stamp: "Formatted at 2026-08-08 06:04:10 on dist-test-slave-zkpd"
server_key: "1c1dad987d4e55453be7df4ae8dfa43c"
server_key_iv: "71d4b65f07ba8c4bd15480f881ce6a26"
server_key_version: "encryptionkey@0"
I20260808 06:04:10.696941 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "42ac70082f2141ba94295b5345e45357"
format_stamp: "Formatted at 2026-08-08 06:04:10 on dist-test-slave-zkpd"
server_key: "1c1dad987d4e55453be7df4ae8dfa43c"
server_key_iv: "71d4b65f07ba8c4bd15480f881ce6a26"
server_key_version: "encryptionkey@0"
I20260808 06:04:10.700547 4080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:04:10.702803 4096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:10.704073 4080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260808 06:04:10.704207 4080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "42ac70082f2141ba94295b5345e45357"
format_stamp: "Formatted at 2026-08-08 06:04:10 on dist-test-slave-zkpd"
server_key: "1c1dad987d4e55453be7df4ae8dfa43c"
server_key_iv: "71d4b65f07ba8c4bd15480f881ce6a26"
server_key_version: "encryptionkey@0"
I20260808 06:04:10.704314 4080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:10.728971 4080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:10.732326 4080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:10.732543 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:10.739990 4080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41317
I20260808 06:04:10.740020 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41317 every 8 connection(s)
I20260808 06:04:10.740939 4080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:10.743562 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:10.747438 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4080
I20260808 06:04:10.747586 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:10.747844 14229 external_mini_cluster.cc:1428] Setting key 363787b257647f6f11cdf560c2f58e16
I20260808 06:04:10.749306 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357: Bootstrap starting.
I20260808 06:04:10.752049 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:10.752947 4150 log.cc:824] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:10.754964 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357: No bootstrap required, opened a new log
I20260808 06:04:10.758091 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:10.749304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:33949 (local address 127.0.0.1:41317)
0808 06:04:10.749764 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:10.749782 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:10.749824 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:04:10.750698 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:10.750721 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:10.751140 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:10.751512 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:10.751965 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:10.753441 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:10.754417 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:10.755083 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:10.755237 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:10.756119 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:10.756125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:10.756142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:10.756627 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:10.756693 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:10.756724 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:10.756863 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:10.756990 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 06:04:10.757120 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260808 06:04:10.759454 4150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } }
I20260808 06:04:10.759691 4150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:10.759769 4150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ac70082f2141ba94295b5345e45357, State: Initialized, Role: FOLLOWER
I20260808 06:04:10.760250 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } }
I20260808 06:04:10.760383 4150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:10.760475 4150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:10.760581 4150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:10.761089 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:10.761659 4150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } }
I20260808 06:04:10.762001 4150 leader_election.cc:302] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42ac70082f2141ba94295b5345e45357; no voters:
I20260808 06:04:10.762339 4150 leader_election.cc:288] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:10.762411 4157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:10.762647 4157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [term 1 LEADER]: Becoming Leader. State: Replica: 42ac70082f2141ba94295b5345e45357, State: Running, Role: LEADER
I20260808 06:04:10.762992 4157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } }
I20260808 06:04:10.763571 4150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:10.764672 4160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42ac70082f2141ba94295b5345e45357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } } }
I20260808 06:04:10.764798 4160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:10.765138 4161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ac70082f2141ba94295b5345e45357. Latest consensus state: current_term: 1 leader_uuid: "42ac70082f2141ba94295b5345e45357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ac70082f2141ba94295b5345e45357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41317 } } }
I20260808 06:04:10.765287 4161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:10.766466 4169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:10.768232 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:10.763165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:36313 (local address 127.0.0.1:41317)
0808 06:04:10.763272 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:10.763276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:10.763513 (+ 237us) server_negotiation.cc:407] Connection header received
0808 06:04:10.763630 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:10.763633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:10.763676 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:10.763745 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:10.764020 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:10.764752 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:10.765943 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:10.766363 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:10.766470 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:10.767458 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:10.767462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:10.767465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:10.767477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:10.767508 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:10.767518 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:10.767617 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:10.768042 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 06:04:10.768077 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:04:10.769192 4169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:10.774546 4169 catalog_manager.cc:1347] Generated new cluster ID: f1352cf07f2f455585915b3f8403c63e
I20260808 06:04:10.774611 4169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:10.798584 4169 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:10.799475 4169 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:10.804085 4169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42ac70082f2141ba94295b5345e45357: Generated new TSK 0
I20260808 06:04:10.804682 4169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:10.817467 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4080
W20260808 06:04:10.825491 4065 connection.cc:535] client connection to 127.0.0.1:41317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41317): Connection reset by peer (error 104)
2026-08-08T06:04:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:10 dist-test-slave-zkpd krb5kdc[4181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:10 dist-test-slave-zkpd krb5kdc[4181](info): set up 2 sockets
Aug 08 06:04:10 dist-test-slave-zkpd krb5kdc[4181](info): commencing operation
krb5kdc: starting...
W20260808 06:04:11.881059 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:11 dist-test-slave-zkpd krb5kdc[4181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 06:04:12.006534 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:12.121555 4194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:12.121809 4194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:12.121898 4194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:12.125896 4194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:12.125959 4194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:12.125979 4194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:12.125996 4194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:12.126010 4194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:12.131086 4194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:12.132107 4194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:12.133070 4194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:12.141577 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:12.141577 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:12.141599 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:12.141857 4194 server_base.cc:1034] running on GCE node
I20260808 06:04:12.142457 4194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:12.143502 4194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:12.144661 4194 hybrid_clock.cc:648] HybridClock initialized: now 1786169052144643 us; error 31 us; skew 500 ppm
Aug 08 06:04:12 dist-test-slave-zkpd krb5kdc[4181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:12.151726 4194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 06:04:12.152643 4194 webserver.cc:460] Webserver started at http://127.0.0.1:35747/ using document root <none> and password file <none>
I20260808 06:04:12.153151 4194 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:12.153210 4194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:12.153370 4194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:12.155342 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "95ce73934d5b44b09bd7209349bc6783"
format_stamp: "Formatted at 2026-08-08 06:04:12 on dist-test-slave-zkpd"
server_key: "067351ab0a2eb144bc792cf539db610c"
server_key_iv: "4149726ecff4f24da952ec82f07070ad"
server_key_version: "encryptionkey@0"
I20260808 06:04:12.155834 4194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "95ce73934d5b44b09bd7209349bc6783"
format_stamp: "Formatted at 2026-08-08 06:04:12 on dist-test-slave-zkpd"
server_key: "067351ab0a2eb144bc792cf539db610c"
server_key_iv: "4149726ecff4f24da952ec82f07070ad"
server_key_version: "encryptionkey@0"
I20260808 06:04:12.159458 4194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 06:04:12.161753 4209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:12.162966 4194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 06:04:12.163110 4194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "95ce73934d5b44b09bd7209349bc6783"
format_stamp: "Formatted at 2026-08-08 06:04:12 on dist-test-slave-zkpd"
server_key: "067351ab0a2eb144bc792cf539db610c"
server_key_iv: "4149726ecff4f24da952ec82f07070ad"
server_key_version: "encryptionkey@0"
I20260808 06:04:12.163216 4194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:12.167659 4194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:12.170487 4194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:12.170687 4194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:12.177678 4194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43157
I20260808 06:04:12.177738 4263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43157 every 8 connection(s)
I20260808 06:04:12.178589 4194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:12.181298 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:12.182837 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4194
I20260808 06:04:12.182988 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:12.183261 14229 external_mini_cluster.cc:1428] Setting key 2c597b8120049b6e965306df13f14b26
I20260808 06:04:12.187651 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783: Bootstrap starting.
I20260808 06:04:12.188423 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:12.184605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:42253 (local address 127.0.0.1:43157)
0808 06:04:12.185004 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:12.185020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:12.185055 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:04:12.185691 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:12.185708 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:12.185987 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:12.186255 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:12.187111 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:12.187116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:12.187128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:12.187138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:12.187192 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:12.187210 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:12.187318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:12.187442 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 06:04:12.187615 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260808 06:04:12.190207 4264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:12.191047 4264 log.cc:824] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:12.191264 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:12.193094 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783: No bootstrap required, opened a new log
I20260808 06:04:12.195269 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:12.193526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:40077 (local address 127.0.0.1:43157)
0808 06:04:12.193629 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:12.193633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:12.193897 (+ 264us) server_negotiation.cc:407] Connection header received
0808 06:04:12.194051 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:12.194054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:12.194085 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:12.194173 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:12.194892 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:12.194895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:12.194898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:12.194909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:12.194935 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:12.194953 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:12.195041 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:12.195117 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 06:04:12.195157 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 06:04:12.196597 4264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260808 06:04:12.196805 4264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:12.196889 4264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ce73934d5b44b09bd7209349bc6783, State: Initialized, Role: FOLLOWER
I20260808 06:04:12.197341 4264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260808 06:04:12.197492 4264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:12.197582 4264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:12.197701 4264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:12.198556 4264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260808 06:04:12.198891 4264 leader_election.cc:302] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95ce73934d5b44b09bd7209349bc6783; no voters:
I20260808 06:04:12.199199 4264 leader_election.cc:288] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:12.199322 4274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:12.199561 4274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [term 1 LEADER]: Becoming Leader. State: Replica: 95ce73934d5b44b09bd7209349bc6783, State: Running, Role: LEADER
I20260808 06:04:12.199951 4274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } }
I20260808 06:04:12.200452 4264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:12.201815 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95ce73934d5b44b09bd7209349bc6783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260808 06:04:12.201850 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ce73934d5b44b09bd7209349bc6783. Latest consensus state: current_term: 1 leader_uuid: "95ce73934d5b44b09bd7209349bc6783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce73934d5b44b09bd7209349bc6783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43157 } } }
I20260808 06:04:12.201957 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:12.201969 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:12.202394 4283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:12.205825 4283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:12.211548 4283 catalog_manager.cc:1347] Generated new cluster ID: f2f4ab9e4fac47e998da7c56d1ea7b90
I20260808 06:04:12.211621 4283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:12.229148 4283 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:12.229544 4283 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:12.237536 4283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ce73934d5b44b09bd7209349bc6783: Generated new TSK 0
I20260808 06:04:12.238299 4283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:12.244341 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4194
2026-08-08T06:04:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:12 dist-test-slave-zkpd krb5kdc[4295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:12 dist-test-slave-zkpd krb5kdc[4295](info): set up 2 sockets
Aug 08 06:04:12 dist-test-slave-zkpd krb5kdc[4295](info): commencing operation
krb5kdc: starting...
W20260808 06:04:13.327858 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:13.463536 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37761
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:13.579985 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:13.580241 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:13.580317 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:13.584297 4308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:13.584350 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:13.584369 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:13.584386 4308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:13.584401 4308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:13.589520 4308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:37761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:13.590583 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:13.591593 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:13.599798 4316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:13.599803 4314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:13.599814 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:13.600140 4308 server_base.cc:1034] running on GCE node
I20260808 06:04:13.600659 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:13.601740 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:13.602939 4308 hybrid_clock.cc:648] HybridClock initialized: now 1786169053602913 us; error 36 us; skew 500 ppm
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169053, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:13.605710 4308 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:13.606547 4308 webserver.cc:460] Webserver started at http://127.13.229.126:37873/ using document root <none> and password file <none>
I20260808 06:04:13.607049 4308 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:13.607105 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:13.607259 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:13.609264 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "05d90a52e9584f47960479b07f1a9a63"
format_stamp: "Formatted at 2026-08-08 06:04:13 on dist-test-slave-zkpd"
server_key: "1546ddb07facdcad1952088c6e70e39a"
server_key_iv: "bc7424d09bc69275ced522ec48ad9fa4"
server_key_version: "encryptionkey@0"
I20260808 06:04:13.609781 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "05d90a52e9584f47960479b07f1a9a63"
format_stamp: "Formatted at 2026-08-08 06:04:13 on dist-test-slave-zkpd"
server_key: "1546ddb07facdcad1952088c6e70e39a"
server_key_iv: "bc7424d09bc69275ced522ec48ad9fa4"
server_key_version: "encryptionkey@0"
I20260808 06:04:13.613299 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:04:13.615531 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:13.616762 4308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:04:13.616918 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "05d90a52e9584f47960479b07f1a9a63"
format_stamp: "Formatted at 2026-08-08 06:04:13 on dist-test-slave-zkpd"
server_key: "1546ddb07facdcad1952088c6e70e39a"
server_key_iv: "bc7424d09bc69275ced522ec48ad9fa4"
server_key_version: "encryptionkey@0"
I20260808 06:04:13.617010 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:13.625756 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:13.629128 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:13.629278 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:13.636456 4308 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:37761
I20260808 06:04:13.636466 4375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:37761 every 8 connection(s)
I20260808 06:04:13.637387 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:13.640130 4376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:13.640702 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4308
I20260808 06:04:13.640863 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:13.641182 14229 external_mini_cluster.cc:1428] Setting key 3f6cf79a5586f687337822a6445ac9b0
I20260808 06:04:13.646610 4376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: Bootstrap starting.
I20260808 06:04:13.648787 4376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:13.649493 4376 log.cc:824] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: Log is configured to *not* fsync() on all Append() calls
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:13.651366 4376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: No bootstrap required, opened a new log
I20260808 06:04:13.654807 4376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } }
I20260808 06:04:13.655027 4376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:13.655095 4376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d90a52e9584f47960479b07f1a9a63, State: Initialized, Role: FOLLOWER
I20260808 06:04:13.655435 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:13.642611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39085 (local address 127.13.229.126:37761)
0808 06:04:13.643024 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:13.643046 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:13.643091 (+ 45us) server_negotiation.cc:407] Connection header received
0808 06:04:13.643947 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:13.643970 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:13.644383 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:13.644738 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:13.645420 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:13.646483 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:13.647400 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:13.647867 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:13.648103 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:13.650645 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:13.650650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:13.650663 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:13.650690 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:13.652392 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:13.652920 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:13.652925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:13.652929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:13.653035 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:13.653387 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:13.653398 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:13.653399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:13.653782 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:13.654015 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:13.654358 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 06:04:13.654573 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260808 06:04:13.655544 4376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } }
I20260808 06:04:13.655748 4376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:13.655843 4376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:13.655982 4376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:13.656986 4376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } }
I20260808 06:04:13.657330 4376 leader_election.cc:302] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d90a52e9584f47960479b07f1a9a63; no voters:
I20260808 06:04:13.657689 4376 leader_election.cc:288] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:13.657797 4381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:13.658020 4381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [term 1 LEADER]: Becoming Leader. State: Replica: 05d90a52e9584f47960479b07f1a9a63, State: Running, Role: LEADER
I20260808 06:04:13.658383 4381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } }
I20260808 06:04:13.658576 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:13.658950 4376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:13.660154 4383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d90a52e9584f47960479b07f1a9a63. Latest consensus state: current_term: 1 leader_uuid: "05d90a52e9584f47960479b07f1a9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } } }
I20260808 06:04:13.660300 4383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:13.660154 4382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d90a52e9584f47960479b07f1a9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d90a52e9584f47960479b07f1a9a63" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 37761 } } }
I20260808 06:04:13.660362 4382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63 [sys.catalog]: This master's current role is: LEADER
W20260808 06:04:13.663965 4400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:13.664054 4400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:13.664160 4396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:13.665040 4396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:13.671186 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:13.661482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60317 (local address 127.13.229.126:37761)
0808 06:04:13.661602 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:13.661608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:13.662269 (+ 661us) server_negotiation.cc:407] Connection header received
0808 06:04:13.662657 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:13.662660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:13.662706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:13.662793 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:13.663598 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:13.664383 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:13.665475 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:13.665846 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:13.665964 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:13.667493 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:13.667497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:13.667500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:13.667524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:13.669123 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:13.669756 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:13.669760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:13.669762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:13.669849 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:13.670228 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:13.670232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:13.670233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:13.670439 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:13.670587 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:13.670944 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 06:04:13.671044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 06:04:13.671406 4396 catalog_manager.cc:1347] Generated new cluster ID: 5d5da6efc37048f5ad34ca95b0d2ebf1
I20260808 06:04:13.671478 4396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:13.695530 4396 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:13.696496 4396 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:13.704663 4396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d90a52e9584f47960479b07f1a9a63: Generated new TSK 0
I20260808 06:04:13.705334 4396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:13.716637 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4308
W20260808 06:04:13.724247 4293 connection.cc:535] client connection to 127.13.229.126:37761 recv error: Network error: failed to read from TLS socket (remote: 127.13.229.126:37761): Connection reset by peer (error 104)
2026-08-08T06:04:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4408](info): set up 2 sockets
Aug 08 06:04:13 dist-test-slave-zkpd krb5kdc[4408](info): commencing operation
krb5kdc: starting...
W20260808 06:04:14.787858 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:14 dist-test-slave-zkpd krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:14.916847 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:46527
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:15.030922 4421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:15.031275 4421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:15.031365 4421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:15.035485 4421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:15.035596 4421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:15.035638 4421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:15.035697 4421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:15.035738 4421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:15.041119 4421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:46527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:15.042256 4421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:15.043326 4421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:15.052049 4427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:15.052026 4429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:15.052145 4421 server_base.cc:1034] running on GCE node
W20260808 06:04:15.052049 4426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:15.052876 4421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:15.053980 4421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:15.055169 4421 hybrid_clock.cc:648] HybridClock initialized: now 1786169055055141 us; error 41 us; skew 500 ppm
Aug 08 06:04:15 dist-test-slave-zkpd krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169055, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:15.058020 4421 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:15.058821 4421 webserver.cc:460] Webserver started at http://127.13.229.126:37067/ using document root <none> and password file <none>
I20260808 06:04:15.059322 4421 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:15.059381 4421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:15.059538 4421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:15.061525 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "4a63331233724d15969fb489474c634b"
format_stamp: "Formatted at 2026-08-08 06:04:15 on dist-test-slave-zkpd"
server_key: "ad3d44ad345bac66096ec550d193a80b"
server_key_iv: "40af52622119bdc1c2534ebfe946f992"
server_key_version: "encryptionkey@0"
I20260808 06:04:15.062065 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "4a63331233724d15969fb489474c634b"
format_stamp: "Formatted at 2026-08-08 06:04:15 on dist-test-slave-zkpd"
server_key: "ad3d44ad345bac66096ec550d193a80b"
server_key_iv: "40af52622119bdc1c2534ebfe946f992"
server_key_version: "encryptionkey@0"
I20260808 06:04:15.065622 4421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260808 06:04:15.067866 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:15.069128 4421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:04:15.069226 4421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "4a63331233724d15969fb489474c634b"
format_stamp: "Formatted at 2026-08-08 06:04:15 on dist-test-slave-zkpd"
server_key: "ad3d44ad345bac66096ec550d193a80b"
server_key_iv: "40af52622119bdc1c2534ebfe946f992"
server_key_version: "encryptionkey@0"
I20260808 06:04:15.069325 4421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:15.080039 4421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:15.083067 4421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:15.083226 4421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:15.090193 4421 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:46527
I20260808 06:04:15.090222 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:46527 every 8 connection(s)
I20260808 06:04:15.091113 4421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:15.093928 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:15.094298 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4421
I20260808 06:04:15.094444 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:15.094764 14229 external_mini_cluster.cc:1428] Setting key 87176e871e71864c2344ef7afbb98221
I20260808 06:04:15.100455 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b: Bootstrap starting.
I20260808 06:04:15.103029 4491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:15.103760 4491 log.cc:824] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:15.104061 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:15.096210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55547 (local address 127.13.229.126:46527)
0808 06:04:15.096633 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:15.096647 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:15.096682 (+ 35us) server_negotiation.cc:407] Connection header received
0808 06:04:15.097342 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:15.097358 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:15.097683 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:15.098032 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:15.098428 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:15.099374 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:15.100377 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:15.101026 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:15.101191 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:15.102357 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:15.102362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:15.102376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:15.102737 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:15.102787 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:15.102809 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:15.102962 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:15.103112 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 06:04:15.103255 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260808 06:04:15.105643 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b: No bootstrap required, opened a new log
I20260808 06:04:15.106985 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:15.109530 4491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } }
I20260808 06:04:15.109835 4491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:15.109915 4491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a63331233724d15969fb489474c634b, State: Initialized, Role: FOLLOWER
I20260808 06:04:15.110414 4491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } }
I20260808 06:04:15.110541 4491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:15.110630 4491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:15.110739 4491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:15.111828 4491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } }
I20260808 06:04:15.112267 4491 leader_election.cc:302] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a63331233724d15969fb489474c634b; no voters:
I20260808 06:04:15.112617 4491 leader_election.cc:288] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:15.112715 4501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:15.112989 4501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [term 1 LEADER]: Becoming Leader. State: Replica: 4a63331233724d15969fb489474c634b, State: Running, Role: LEADER
I20260808 06:04:15.113332 4501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } }
I20260808 06:04:15.113633 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:15.109090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48655 (local address 127.13.229.126:46527)
0808 06:04:15.109222 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:15.109227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:15.109410 (+ 183us) server_negotiation.cc:407] Connection header received
0808 06:04:15.109548 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:15.109550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:15.109584 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:15.109648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:15.109977 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:15.110578 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:15.111715 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:15.112038 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:15.112133 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:15.113114 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:15.113118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:15.113121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:15.113133 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:15.113160 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:15.113172 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:15.113228 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:15.113480 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 06:04:15.113516 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 06:04:15.113994 4491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:15.115059 4503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a63331233724d15969fb489474c634b. Latest consensus state: current_term: 1 leader_uuid: "4a63331233724d15969fb489474c634b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } } }
I20260808 06:04:15.115195 4503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:15.115528 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a63331233724d15969fb489474c634b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a63331233724d15969fb489474c634b" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 46527 } } }
I20260808 06:04:15.115608 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:15.115614 4510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:15.118572 4510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:15.124382 4510 catalog_manager.cc:1347] Generated new cluster ID: a43a0b6d71cd4ecea7f2f444a6946818
I20260808 06:04:15.124450 4510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:15.158466 4510 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:15.159705 4510 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:15.168229 4510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a63331233724d15969fb489474c634b: Generated new TSK 0
I20260808 06:04:15.169108 4510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:15.183926 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4421
W20260808 06:04:15.192107 4406 connection.cc:535] client connection to 127.13.229.126:46527 recv error: Network error: failed to read from TLS socket (remote: 127.13.229.126:46527): Connection reset by peer (error 104)
2026-08-08T06:04:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:15 dist-test-slave-zkpd krb5kdc[4522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:15 dist-test-slave-zkpd krb5kdc[4522](info): set up 2 sockets
Aug 08 06:04:15 dist-test-slave-zkpd krb5kdc[4522](info): commencing operation
krb5kdc: starting...
W20260808 06:04:16.322268 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:16 dist-test-slave-zkpd krb5kdc[4522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:16.446990 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41483
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:41483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:16.556691 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:16.556968 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:16.557041 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:16.560920 4535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:16.561002 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:16.561039 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:16.561086 4535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:16.561122 4535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:16.566179 4535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:41483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:41483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:16.567152 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:16.568140 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:16.576203 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:16.576327 4535 server_base.cc:1034] running on GCE node
W20260808 06:04:16.576218 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:16.576210 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:16.576972 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:16.577975 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:16.579166 4535 hybrid_clock.cc:648] HybridClock initialized: now 1786169056579150 us; error 33 us; skew 500 ppm
Aug 08 06:04:16 dist-test-slave-zkpd krb5kdc[4522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169056, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:16.581840 4535 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:16.582775 4535 webserver.cc:460] Webserver started at http://127.13.229.126:40163/ using document root <none> and password file <none>
I20260808 06:04:16.583320 4535 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:16.583412 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:16.583634 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:16.585647 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "3c52e03b57a641e8a79af1632a2af484"
format_stamp: "Formatted at 2026-08-08 06:04:16 on dist-test-slave-zkpd"
server_key: "8ccf6b13491697a9beb6bf5fd2cf8efb"
server_key_iv: "0453105a9cee2555853731af74cbfeea"
server_key_version: "encryptionkey@0"
I20260808 06:04:16.586210 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "3c52e03b57a641e8a79af1632a2af484"
format_stamp: "Formatted at 2026-08-08 06:04:16 on dist-test-slave-zkpd"
server_key: "8ccf6b13491697a9beb6bf5fd2cf8efb"
server_key_iv: "0453105a9cee2555853731af74cbfeea"
server_key_version: "encryptionkey@0"
I20260808 06:04:16.589808 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:04:16.592218 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:16.593456 4535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:04:16.593604 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "3c52e03b57a641e8a79af1632a2af484"
format_stamp: "Formatted at 2026-08-08 06:04:16 on dist-test-slave-zkpd"
server_key: "8ccf6b13491697a9beb6bf5fd2cf8efb"
server_key_iv: "0453105a9cee2555853731af74cbfeea"
server_key_version: "encryptionkey@0"
I20260808 06:04:16.593713 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:16.598203 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:16.601003 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:16.601191 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:16.608438 4535 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:41483
I20260808 06:04:16.608469 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:41483 every 8 connection(s)
I20260808 06:04:16.609324 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:16.611971 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:16.612846 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4535
I20260808 06:04:16.613000 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:16.613291 14229 external_mini_cluster.cc:1428] Setting key a6e54139633cbd83949c9575f8e5a4d1
I20260808 06:04:16.618809 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484: Bootstrap starting.
I20260808 06:04:16.618957 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:16.614664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53043 (local address 127.13.229.126:41483)
0808 06:04:16.615111 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:16.615126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:16.615167 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:04:16.615824 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:16.615849 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:16.616245 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:16.616662 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:16.617775 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:16.617781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:16.617802 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:16.617813 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:16.617867 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:16.617891 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:16.618044 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:16.618114 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 06:04:16.618270 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260808 06:04:16.621454 4607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:16.621783 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:16.622365 4607 log.cc:824] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:16.624512 4607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484: No bootstrap required, opened a new log
I20260808 06:04:16.625867 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:16.624106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56431 (local address 127.13.229.126:41483)
0808 06:04:16.624225 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:16.624229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:16.624604 (+ 375us) server_negotiation.cc:407] Connection header received
0808 06:04:16.624659 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:16.624667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:16.624698 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:16.624761 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:16.625459 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:16.625462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:16.625464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:16.625474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:16.625498 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:16.625508 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:16.625587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:16.625700 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 06:04:16.625729 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 06:04:16.628799 4607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } }
I20260808 06:04:16.629011 4607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:16.629101 4607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c52e03b57a641e8a79af1632a2af484, State: Initialized, Role: FOLLOWER
I20260808 06:04:16.629592 4607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } }
I20260808 06:04:16.629751 4607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:16.629825 4607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:16.629925 4607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:16.630797 4607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } }
I20260808 06:04:16.631141 4607 leader_election.cc:302] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c52e03b57a641e8a79af1632a2af484; no voters:
I20260808 06:04:16.631446 4607 leader_election.cc:288] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:16.631572 4617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:16.631829 4617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [term 1 LEADER]: Becoming Leader. State: Replica: 3c52e03b57a641e8a79af1632a2af484, State: Running, Role: LEADER
I20260808 06:04:16.632233 4617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } }
I20260808 06:04:16.632671 4607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:16.633962 4619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c52e03b57a641e8a79af1632a2af484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } } }
I20260808 06:04:16.634111 4619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:16.634446 4620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c52e03b57a641e8a79af1632a2af484. Latest consensus state: current_term: 1 leader_uuid: "3c52e03b57a641e8a79af1632a2af484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c52e03b57a641e8a79af1632a2af484" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 41483 } } }
I20260808 06:04:16.634541 4620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:16.634639 4628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:16.637892 4628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:16.643504 4628 catalog_manager.cc:1347] Generated new cluster ID: 4fc646c20f86457289d2def4ffe8067c
I20260808 06:04:16.643568 4628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:16.679677 4628 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:16.680114 4628 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:16.685446 4628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c52e03b57a641e8a79af1632a2af484: Generated new TSK 0
I20260808 06:04:16.686242 4628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:16.692971 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4535
2026-08-08T06:04:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:16 dist-test-slave-zkpd krb5kdc[4641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:16 dist-test-slave-zkpd krb5kdc[4641](info): set up 2 sockets
Aug 08 06:04:16 dist-test-slave-zkpd krb5kdc[4641](info): commencing operation
krb5kdc: starting...
W20260808 06:04:17.784879 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:17 dist-test-slave-zkpd krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:17.909250 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38367
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:18.021632 4655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:18.021888 4655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:18.021927 4655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:18.025739 4655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:18.025786 4655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:18.025805 4655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:18.025821 4655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:18.025836 4655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:18.030797 4655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:18.031778 4655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:18.032758 4655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:18.041000 4660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:18.041008 4664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:18.041054 4662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:18.041146 4655 server_base.cc:1034] running on GCE node
I20260808 06:04:18.041749 4655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:18.042752 4655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:18.043960 4655 hybrid_clock.cc:648] HybridClock initialized: now 1786169058043942 us; error 29 us; skew 500 ppm
Aug 08 06:04:18 dist-test-slave-zkpd krb5kdc[4641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169058, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:18.051636 4655 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:18.052597 4655 webserver.cc:460] Webserver started at http://127.13.229.126:41253/ using document root <none> and password file <none>
I20260808 06:04:18.053153 4655 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:18.053213 4655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:18.053447 4655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:18.055668 4655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e"
format_stamp: "Formatted at 2026-08-08 06:04:18 on dist-test-slave-zkpd"
server_key: "ad75a7547ec65c335b7a8e1380118e33"
server_key_iv: "cec3788d3dbe8e121c61743fb5cd5784"
server_key_version: "encryptionkey@0"
I20260808 06:04:18.056303 4655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e"
format_stamp: "Formatted at 2026-08-08 06:04:18 on dist-test-slave-zkpd"
server_key: "ad75a7547ec65c335b7a8e1380118e33"
server_key_iv: "cec3788d3dbe8e121c61743fb5cd5784"
server_key_version: "encryptionkey@0"
I20260808 06:04:18.059957 4655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:04:18.062251 4672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:18.063448 4655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 06:04:18.063591 4655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e"
format_stamp: "Formatted at 2026-08-08 06:04:18 on dist-test-slave-zkpd"
server_key: "ad75a7547ec65c335b7a8e1380118e33"
server_key_iv: "cec3788d3dbe8e121c61743fb5cd5784"
server_key_version: "encryptionkey@0"
I20260808 06:04:18.063696 4655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:18.074822 4655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:18.077657 4655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:18.077847 4655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:18.084892 4655 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:38367
I20260808 06:04:18.084969 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:38367 every 8 connection(s)
I20260808 06:04:18.085862 4655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:18.088621 4725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:18.094437 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: Bootstrap starting.
I20260808 06:04:18.095952 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4655
I20260808 06:04:18.096122 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:18.096393 14229 external_mini_cluster.cc:1428] Setting key 875f8d7e54ec76197150a439aa3ba419
I20260808 06:04:18.097378 4725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:18.098387 4725 log.cc:824] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:18.100721 4725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: No bootstrap required, opened a new log
I20260808 06:04:18.105510 4725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } }
I20260808 06:04:18.105785 4725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:18.105878 4725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0f973666eb4d8dbea5d8ef0b4a692e, State: Initialized, Role: FOLLOWER
Aug 08 06:04:18 dist-test-slave-zkpd krb5kdc[4641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:18.106447 4725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } }
I20260808 06:04:18.106613 4725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:18.106694 4725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:18.106820 4725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:18.108222 4725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } }
I20260808 06:04:18.108625 4725 leader_election.cc:302] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c0f973666eb4d8dbea5d8ef0b4a692e; no voters:
I20260808 06:04:18.109072 4725 leader_election.cc:288] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:18.109172 4730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:18.109436 4730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [term 1 LEADER]: Becoming Leader. State: Replica: 2c0f973666eb4d8dbea5d8ef0b4a692e, State: Running, Role: LEADER
I20260808 06:04:18.109797 4730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } }
I20260808 06:04:18.110638 4725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:18.112368 4732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c0f973666eb4d8dbea5d8ef0b4a692e. Latest consensus state: current_term: 1 leader_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } } }
I20260808 06:04:18.112473 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0f973666eb4d8dbea5d8ef0b4a692e" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 38367 } } }
I20260808 06:04:18.112555 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:18.112511 4732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:18.113631 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:18.097501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35659 (local address 127.13.229.126:38367)
0808 06:04:18.097994 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:18.098013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:18.098057 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:04:18.098984 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:18.099005 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:18.099453 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:18.099827 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:18.100607 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:18.102300 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:18.103150 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:18.104048 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:18.104276 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:18.106854 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:18.106861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:18.106879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:18.106919 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:18.109180 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:18.109787 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:18.109793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:18.109799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:18.109977 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:18.110355 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:18.110358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:18.110360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:18.110901 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:18.111882 (+ 981us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:18.112086 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 06:04:18.112548 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":148,"threads_started":1}
I20260808 06:04:18.117043 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 06:04:18.117384 4746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:18.117456 4746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:18.117543 4735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:18.118464 4735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:18.125391 4735 catalog_manager.cc:1347] Generated new cluster ID: 1759bc6562af43d282a8d9c9acec0f3e
I20260808 06:04:18.125463 4735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:18.130157 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:18.119402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.13.229.126:38367)
0808 06:04:18.119529 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:18.119534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:18.119959 (+ 425us) server_negotiation.cc:407] Connection header received
0808 06:04:18.120109 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:18.120113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:18.120165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:18.120262 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:18.120974 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:18.121504 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:18.122300 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:18.122557 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:18.122674 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:18.124378 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:18.124383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:18.124386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:18.124414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:18.128489 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:18.128930 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:18.128934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:18.128936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:18.129021 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:18.129298 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:18.129301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:18.129303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:18.129488 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:18.129643 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:18.129876 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 06:04:18.129996 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:04:18.134796 4735 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:18.136065 4735 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:18.142774 4735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c0f973666eb4d8dbea5d8ef0b4a692e: Generated new TSK 0
I20260808 06:04:18.143544 4735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:18.153028 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4655
2026-08-08T06:04:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:18 dist-test-slave-zkpd krb5kdc[4757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:18 dist-test-slave-zkpd krb5kdc[4757](info): set up 2 sockets
Aug 08 06:04:18 dist-test-slave-zkpd krb5kdc[4757](info): commencing operation
krb5kdc: starting...
W20260808 06:04:19.217583 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:19 dist-test-slave-zkpd krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:19.346560 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36059
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:19.455168 4771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:19.455412 4771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:19.455453 4771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:19.460631 4771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:19.460704 4771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:19.460737 4771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:19.460762 4771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:19.460795 4771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:19.468523 4771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:36059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:19.469594 4771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:19.470613 4771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:19.479288 4779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:19.479442 4771 server_base.cc:1034] running on GCE node
W20260808 06:04:19.479310 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:19.479288 4777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:19.480197 4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:19.481284 4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:19.482450 4771 hybrid_clock.cc:648] HybridClock initialized: now 1786169059482432 us; error 39 us; skew 500 ppm
Aug 08 06:04:19 dist-test-slave-zkpd krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169059, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:19.485303 4771 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:19.486241 4771 webserver.cc:460] Webserver started at http://127.13.229.126:43703/ using document root <none> and password file <none>
I20260808 06:04:19.486804 4771 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:19.486898 4771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:19.487150 4771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:19.489302 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "7a5ab712731649b8807e0b636f5b258c"
format_stamp: "Formatted at 2026-08-08 06:04:19 on dist-test-slave-zkpd"
server_key: "88b24e19a6005e435400b3a3c96a6d5c"
server_key_iv: "d36b3ebcdebdb900b24608ee68efc119"
server_key_version: "encryptionkey@0"
I20260808 06:04:19.489881 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "7a5ab712731649b8807e0b636f5b258c"
format_stamp: "Formatted at 2026-08-08 06:04:19 on dist-test-slave-zkpd"
server_key: "88b24e19a6005e435400b3a3c96a6d5c"
server_key_iv: "d36b3ebcdebdb900b24608ee68efc119"
server_key_version: "encryptionkey@0"
I20260808 06:04:19.493592 4771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:04:19.496053 4786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:19.497215 4771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:04:19.497367 4771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "7a5ab712731649b8807e0b636f5b258c"
format_stamp: "Formatted at 2026-08-08 06:04:19 on dist-test-slave-zkpd"
server_key: "88b24e19a6005e435400b3a3c96a6d5c"
server_key_iv: "d36b3ebcdebdb900b24608ee68efc119"
server_key_version: "encryptionkey@0"
I20260808 06:04:19.497481 4771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:19.525557 4771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:19.528489 4771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:19.528714 4771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:19.535976 4771 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:36059
I20260808 06:04:19.536002 4840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:36059 every 8 connection(s)
I20260808 06:04:19.536876 4771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:19.539553 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:19.542680 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4771
I20260808 06:04:19.542843 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:19.543192 14229 external_mini_cluster.cc:1428] Setting key a29864338c2a74697e2a9989e3404776
I20260808 06:04:19.545615 4841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: Bootstrap starting.
I20260808 06:04:19.547760 4841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:19.548522 4841 log.cc:824] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:19.550809 4841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: No bootstrap required, opened a new log
I20260808 06:04:19.552611 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:19.544629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50133 (local address 127.13.229.126:36059)
0808 06:04:19.545082 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:19.545097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:19.545131 (+ 34us) server_negotiation.cc:407] Connection header received
0808 06:04:19.545812 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:19.545831 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:19.546156 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:19.546493 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:19.546901 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:19.548007 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:19.549000 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:19.549665 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:19.549885 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:19.550655 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:19.550661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:19.550678 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:19.551183 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:19.551251 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:19.551282 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:19.551382 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:19.551488 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 06:04:19.551688 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":138,"threads_started":1}
I20260808 06:04:19.555200 4841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } }
I20260808 06:04:19.555476 4841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:19.555598 4841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5ab712731649b8807e0b636f5b258c, State: Initialized, Role: FOLLOWER
I20260808 06:04:19.555686 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:19.556107 4841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } }
I20260808 06:04:19.556262 4841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:19.556331 4841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:19.556422 4841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:19.557380 4841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } }
I20260808 06:04:19.557713 4841 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5ab712731649b8807e0b636f5b258c; no voters:
I20260808 06:04:19.558110 4841 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:19.558495 4851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:19.558769 4851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [term 1 LEADER]: Becoming Leader. State: Replica: 7a5ab712731649b8807e0b636f5b258c, State: Running, Role: LEADER
I20260808 06:04:19.559119 4851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } }
I20260808 06:04:19.559495 4841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:19.560901 4852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5ab712731649b8807e0b636f5b258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } } }
I20260808 06:04:19.560900 4853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5ab712731649b8807e0b636f5b258c. Latest consensus state: current_term: 1 leader_uuid: "7a5ab712731649b8807e0b636f5b258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5ab712731649b8807e0b636f5b258c" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 36059 } } }
I20260808 06:04:19.561076 4853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:19.561076 4852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:19.562973 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:19.557914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54467 (local address 127.13.229.126:36059)
0808 06:04:19.558058 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:19.558064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:19.558263 (+ 199us) server_negotiation.cc:407] Connection header received
0808 06:04:19.558349 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:19.558501 (+ 152us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:19.558556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:19.558637 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:19.559028 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:19.559779 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:19.560997 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:19.561393 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:19.561504 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:19.562510 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:19.562513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:19.562516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:19.562528 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:19.562562 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:19.562573 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:19.562655 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:19.562785 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 06:04:19.562836 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 06:04:19.564340 4866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:19.564437 4866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:19.564517 4863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:19.565219 4863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:19.570199 4863 catalog_manager.cc:1347] Generated new cluster ID: e09999eb767443da87b7a193f0195f93
I20260808 06:04:19.570266 4863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:19.592361 4863 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:19.593281 4863 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:19.600368 4863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5ab712731649b8807e0b636f5b258c: Generated new TSK 0
I20260808 06:04:19.601080 4863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:19.609295 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4771
2026-08-08T06:04:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:19 dist-test-slave-zkpd krb5kdc[4872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:19 dist-test-slave-zkpd krb5kdc[4872](info): set up 2 sockets
Aug 08 06:04:19 dist-test-slave-zkpd krb5kdc[4872](info): commencing operation
krb5kdc: starting...
W20260808 06:04:20.682024 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:20 dist-test-slave-zkpd krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:20.810132 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45187
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:45187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:20.920687 4885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:20.920966 4885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:20.921008 4885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:20.924959 4885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:20.925017 4885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:20.925036 4885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:20.925053 4885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:20.925068 4885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:20.930203 4885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:45187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:45187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:20.931205 4885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:20.932266 4885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:20.940732 4890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:20.940753 4891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:20.940753 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:20.941064 4885 server_base.cc:1034] running on GCE node
I20260808 06:04:20.941709 4885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:20.942830 4885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:20.944020 4885 hybrid_clock.cc:648] HybridClock initialized: now 1786169060943991 us; error 40 us; skew 500 ppm
Aug 08 06:04:20 dist-test-slave-zkpd krb5kdc[4872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169060, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:20.946760 4885 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:20.947623 4885 webserver.cc:460] Webserver started at http://127.13.229.126:35071/ using document root <none> and password file <none>
I20260808 06:04:20.948246 4885 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:20.948338 4885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:20.948575 4885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:20.950654 4885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "32020cec9fdc48d5b09eedcdfc7b12b8"
format_stamp: "Formatted at 2026-08-08 06:04:20 on dist-test-slave-zkpd"
server_key: "d30410adfa3e36a2080258396b71f989"
server_key_iv: "beca0b77017d53ba3450d6f2718896c9"
server_key_version: "encryptionkey@0"
I20260808 06:04:20.951236 4885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "32020cec9fdc48d5b09eedcdfc7b12b8"
format_stamp: "Formatted at 2026-08-08 06:04:20 on dist-test-slave-zkpd"
server_key: "d30410adfa3e36a2080258396b71f989"
server_key_iv: "beca0b77017d53ba3450d6f2718896c9"
server_key_version: "encryptionkey@0"
I20260808 06:04:20.954903 4885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 06:04:20.957341 4900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:20.958550 4885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:20.958693 4885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "32020cec9fdc48d5b09eedcdfc7b12b8"
format_stamp: "Formatted at 2026-08-08 06:04:20 on dist-test-slave-zkpd"
server_key: "d30410adfa3e36a2080258396b71f989"
server_key_iv: "beca0b77017d53ba3450d6f2718896c9"
server_key_version: "encryptionkey@0"
I20260808 06:04:20.958806 4885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:20.963534 4885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:20.966431 4885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:20.966624 4885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:20.973562 4885 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:45187
I20260808 06:04:20.973616 4957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:45187 every 8 connection(s)
I20260808 06:04:20.974526 4885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:20.976217 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 4885
I20260808 06:04:20.976357 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:20.976645 14229 external_mini_cluster.cc:1428] Setting key f92e3a87d0141c8822287213415bd3a3
I20260808 06:04:20.977576 4958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:20.982295 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:20.978050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44057 (local address 127.13.229.126:45187)
0808 06:04:20.978487 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:20.978507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:20.978549 (+ 42us) server_negotiation.cc:407] Connection header received
0808 06:04:20.979398 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:20.979423 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:20.979774 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:20.980177 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:20.980863 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:20.980871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:20.980886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:20.980898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:20.980987 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:20.981016 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:20.981162 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:20.981243 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 06:04:20.981394 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260808 06:04:20.984591 4958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8: Bootstrap starting.
I20260808 06:04:20.985034 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:20.986892 4958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:20.987818 4958 log.cc:824] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:20.988737 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:20.987056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42315 (local address 127.13.229.126:45187)
0808 06:04:20.987189 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:20.987194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:20.987385 (+ 191us) server_negotiation.cc:407] Connection header received
0808 06:04:20.987466 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:20.987469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:20.987500 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:20.987581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:20.988328 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:20.988333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:20.988335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 06:04:20.988348 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:20.988387 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 06:04:20.988404 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:20.988497 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:20.988571 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 06:04:20.988611 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 06:04:20.989821 4958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8: No bootstrap required, opened a new log
I20260808 06:04:20.993275 4958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } }
I20260808 06:04:20.993494 4958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 06:04:20.993582 4958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32020cec9fdc48d5b09eedcdfc7b12b8, State: Initialized, Role: FOLLOWER
I20260808 06:04:20.994031 4958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } }
I20260808 06:04:20.994176 4958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 06:04:20.994246 4958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 06:04:20.994393 4958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:20.995249 4958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } }
I20260808 06:04:20.995532 4958 leader_election.cc:302] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32020cec9fdc48d5b09eedcdfc7b12b8; no voters:
I20260808 06:04:20.995780 4958 leader_election.cc:288] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 06:04:20.995919 4968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:20.996194 4968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [term 1 LEADER]: Becoming Leader. State: Replica: 32020cec9fdc48d5b09eedcdfc7b12b8, State: Running, Role: LEADER
I20260808 06:04:20.996549 4968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } }
I20260808 06:04:20.996973 4958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:20.998095 4970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32020cec9fdc48d5b09eedcdfc7b12b8. Latest consensus state: current_term: 1 leader_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } } }
I20260808 06:04:20.998229 4970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:20.998656 4977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:20.998670 4969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32020cec9fdc48d5b09eedcdfc7b12b8" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 45187 } } }
I20260808 06:04:20.998819 4969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:21.001837 4977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:21.007650 4977 catalog_manager.cc:1347] Generated new cluster ID: 8839e34e19df4e46b33ca102608ac1ff
I20260808 06:04:21.007733 4977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:21.030890 4977 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:21.031275 4977 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:21.040475 4977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32020cec9fdc48d5b09eedcdfc7b12b8: Generated new TSK 0
I20260808 06:04:21.041237 4977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 06:04:21.056478 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 4885
2026-08-08T06:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1446 ms)
[----------] 18 tests from AuthTokenIssuingTest (26075 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:21 dist-test-slave-zkpd krb5kdc[4989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:21 dist-test-slave-zkpd krb5kdc[4989](info): set up 2 sockets
Aug 08 06:04:21 dist-test-slave-zkpd krb5kdc[4989](info): commencing operation
krb5kdc: starting...
W20260808 06:04:22.170475 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:22.294759 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33723
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:22.409991 5003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:22.410292 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:22.410319 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:22.410360 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:22.414848 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:22.414906 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:22.414927 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:22.414943 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:22.414958 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:22.420145 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:22.421169 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:22.422137 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:22.430552 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.430559 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.430573 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:22.430860 5003 server_base.cc:1034] running on GCE node
I20260808 06:04:22.431433 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:22.432648 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:22.433861 5003 hybrid_clock.cc:648] HybridClock initialized: now 1786169062433845 us; error 25 us; skew 500 ppm
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:22.436623 5003 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:22.437544 5003 webserver.cc:460] Webserver started at http://127.13.229.126:35911/ using document root <none> and password file <none>
I20260808 06:04:22.438114 5003 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:22.438208 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:22.438470 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:22.440577 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "cc38893cd3674522a4415ea0ca4b4871"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "094a601b33f6b0c96efe38451cea1116"
server_key_iv: "7a87e2c8ca34f3122ccbe329b163e021"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.441159 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "cc38893cd3674522a4415ea0ca4b4871"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "094a601b33f6b0c96efe38451cea1116"
server_key_iv: "7a87e2c8ca34f3122ccbe329b163e021"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.444878 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 06:04:22.447319 5019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:22.448547 5003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 06:04:22.448693 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "cc38893cd3674522a4415ea0ca4b4871"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "094a601b33f6b0c96efe38451cea1116"
server_key_iv: "7a87e2c8ca34f3122ccbe329b163e021"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.448827 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:22.474532 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:22.477481 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:22.477698 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:22.485239 5003 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:33723
I20260808 06:04:22.485526 5075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:33723 every 8 connection(s)
I20260808 06:04:22.486311 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:22.489154 5076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:22.491080 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5003
I20260808 06:04:22.491226 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:22.491549 14229 external_mini_cluster.cc:1428] Setting key 23604a3119dc9ae344d4126f36c03b3c
I20260808 06:04:22.493105 5076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.229.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.125@KRBTEST.COM" created.
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:22.510038 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.495330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33723 (local address 127.0.0.1:57098)
0808 06:04:22.495803 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.495856 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.496715 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.497513 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.497516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.497847 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.498123 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.498135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.499932 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.499938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.500647 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.500655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.501609 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.501612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.501688 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.502271 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.502287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.505040 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.506969 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.506980 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.506989 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.507212 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.507544 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.507547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.507549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.507660 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.508312 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.508318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.508462 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.508623 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.509046 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":144,"thread_start_us":139,"threads_started":1}
I20260808 06:04:22.510038 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.495330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.13.229.126:33723)
0808 06:04:22.495880 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.495915 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.496014 (+ 99us) server_negotiation.cc:407] Connection header received
0808 06:04:22.497031 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.497038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.497296 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.497524 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.498290 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.499762 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.500779 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.501462 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.501595 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.505194 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.505199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.505209 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.505229 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.506812 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.507330 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.507342 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.507343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.507442 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.507757 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.507760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.507761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.508189 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.508326 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.508698 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.508997 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":12672,"thread_start_us":163,"threads_started":1}
I20260808 06:04:22.512691 5076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.513409 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.513159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45648)
0808 06:04:22.513267 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.513336 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:04:22.513680 5024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
I20260808 06:04:22.514567 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.514372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45656)
0808 06:04:22.514476 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.514519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 06:04:22.514751 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:34679: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
I20260808 06:04:22.520211 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:34679
--webserver_interface=127.13.229.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:04:22.564163 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } attempt: 1
I20260808 06:04:22.565124 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.564805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45668)
0808 06:04:22.564981 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.565047 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 06:04:22.565975 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.565763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45680)
0808 06:04:22.565878 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.565923 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:04:22.566109 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:34679: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
W20260808 06:04:22.632303 5084 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:22.632617 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:22.632699 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:22.632777 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 06:04:22.634263 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } attempt: 2
I20260808 06:04:22.635164 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.634878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45684)
0808 06:04:22.635021 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.635090 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 06:04:22.635948 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.635698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45694)
0808 06:04:22.635805 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.635857 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 06:04:22.636083 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:34679: Network error: Client connection negotiation failed: client connection to 127.13.229.125:34679: connect: Connection refused (error 111)
W20260808 06:04:22.637018 5084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:22.637068 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:22.637089 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:22.637147 5084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:22.637185 5084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:22.642330 5084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.229.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:22.643342 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:22.644429 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:22.652583 5093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.652583 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.652606 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:22.652908 5084 server_base.cc:1034] running on GCE node
I20260808 06:04:22.653532 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:22.654518 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:22.655676 5084 hybrid_clock.cc:648] HybridClock initialized: now 1786169062655661 us; error 27 us; skew 500 ppm
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:22.658438 5084 init.cc:377] Logged in from keytab as kudu/127.13.229.125@KRBTEST.COM (short username kudu)
I20260808 06:04:22.659235 5084 webserver.cc:460] Webserver started at http://127.13.229.125:37243/ using document root <none> and password file <none>
I20260808 06:04:22.659709 5084 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:22.659765 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:22.659998 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:22.661880 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data/instance:
uuid: "197f21662874459cae188a0d2673495b"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "348314043dba2878ea5c078f80a6ff0c"
server_key_iv: "67ba137bc24cbde66d527445224a31e4"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.662371 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal/instance:
uuid: "197f21662874459cae188a0d2673495b"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "348314043dba2878ea5c078f80a6ff0c"
server_key_iv: "67ba137bc24cbde66d527445224a31e4"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.665998 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 06:04:22.668273 5100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:22.669464 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:04:22.669585 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "197f21662874459cae188a0d2673495b"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "348314043dba2878ea5c078f80a6ff0c"
server_key_iv: "67ba137bc24cbde66d527445224a31e4"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.669682 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:22.699008 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:22.702075 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:22.702282 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:22.709396 5084 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.125:34679
I20260808 06:04:22.709688 5153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.125:34679 every 8 connection(s)
I20260808 06:04:22.710426 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
I20260808 06:04:22.713147 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:22.716821 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5084
I20260808 06:04:22.716956 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-1/wal/instance
I20260808 06:04:22.717257 14229 external_mini_cluster.cc:1428] Setting key 1ea93e2e17900252c0762da5aa8cd526
I20260808 06:04:22.717034 5154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.229.124@KRBTEST.COM; defaulting to no policy
I20260808 06:04:22.727233 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.124@KRBTEST.COM" created.
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:04:22.735195 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.719215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.13.229.126:33723)
0808 06:04:22.719389 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.719395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.720368 (+ 973us) server_negotiation.cc:407] Connection header received
0808 06:04:22.721296 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.721300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.721365 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.721477 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.722714 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.723252 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.724437 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.724667 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.724780 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.729838 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.729842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.729844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.729878 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.732367 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.733023 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.733029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.733034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.733132 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.733568 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.733574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.733579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.733892 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.734006 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.734811 (+ 805us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.734956 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 06:04:22.735994 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.719662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33723 (local address 127.0.0.1:57104)
0808 06:04:22.720217 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.720261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.721089 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.721647 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.721654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.721997 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.722552 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.722583 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.723399 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.723401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.724301 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.724311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.724820 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.724824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.724994 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.725618 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.725634 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.729681 (+ 4047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.732505 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.732514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.732532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.732886 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.733243 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.733246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.733248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.733443 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.734022 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.734032 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.734314 (+ 282us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.734520 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.734858 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":186,"threads_started":1}
I20260808 06:04:22.737638 5154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.738871 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.727661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45710)
0808 06:04:22.727845 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.727862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.728027 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.728643 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.728647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.728861 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.729030 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.729042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.730158 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.730163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.731047 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.731058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.731814 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.731818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.731971 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.732444 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.732466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.733939 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.737077 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.737080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.737082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.737321 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.737668 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.737672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.737674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.737769 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.738334 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.738337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.738450 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.738541 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.738683 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 06:04:22.739531 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.727789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.13.229.125:34679)
0808 06:04:22.728191 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.728200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.728222 (+ 22us) server_negotiation.cc:407] Connection header received
0808 06:04:22.728259 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.728262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.728430 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.728649 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.729159 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.730012 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.731179 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.731689 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.731815 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.734115 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.734130 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.734145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.734175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.736969 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.737453 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.737460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.737462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.737559 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.737885 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.737888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.737890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.738203 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.739097 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.739209 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.739344 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260808 06:04:22.741559 5076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.742328 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.742101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37182)
0808 06:04:22.742221 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.742269 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:04:22.743198 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.742858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37198)
0808 06:04:22.743079 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.743132 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260808 06:04:22.743340 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.747033 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.738213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45714)
0808 06:04:22.738413 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.738428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.738512 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.738877 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.738880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.739184 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.739306 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.739315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.740231 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.740234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.740915 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.740925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.741528 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.741532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.741647 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.742101 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.742115 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.743567 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.745716 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.745718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.745720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.745910 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.746173 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.746179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.746181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.746267 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.746619 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.746626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.746729 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.746821 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.746927 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260808 06:04:22.747152 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.738303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.13.229.125:34679)
0808 06:04:22.738636 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.738647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.738661 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:04:22.738696 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.738698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.738739 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.738837 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.739460 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.740129 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.741033 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.741404 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.741518 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.743712 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.743715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.743717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.743738 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.745599 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.745994 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.745997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.745998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.746070 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.746350 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.746352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.746353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.746528 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.746628 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.746857 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.746984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
I20260808 06:04:22.747812 5154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.748540 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.748308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37202)
0808 06:04:22.748423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.748491 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 06:04:22.748736 5103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.748747 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:44203
--webserver_interface=127.13.229.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:04:22.749519 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.749316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37206)
0808 06:04:22.749434 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.749474 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 06:04:22.749682 5154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.774528 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 1
I20260808 06:04:22.775309 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.775028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37212)
0808 06:04:22.775166 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.775235 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 06:04:22.776032 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.775798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37216)
0808 06:04:22.775931 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.775981 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 06:04:22.776153 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.798993 5154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 1
I20260808 06:04:22.799928 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.799553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37218)
0808 06:04:22.799731 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.799811 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 06:04:22.800693 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.800445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37228)
0808 06:04:22.800570 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.800625 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 06:04:22.800841 5154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.851418 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 2
I20260808 06:04:22.852352 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.852009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37234)
0808 06:04:22.852159 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.852270 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 06:04:22.853123 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.852951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37236)
0808 06:04:22.853033 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.853077 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 06:04:22.853331 5076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
W20260808 06:04:22.862941 5163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:22.863191 5163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:22.863214 5163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:22.863251 5163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:22.867269 5163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:22.867318 5163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:22.867338 5163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:22.867405 5163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:22.867430 5163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:22.869025 5154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 2
I20260808 06:04:22.869807 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.869492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37242)
0808 06:04:22.869656 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.869721 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 06:04:22.870553 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.870327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37250)
0808 06:04:22.870442 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.870493 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 06:04:22.870723 5154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:44203: Network error: Client connection negotiation failed: client connection to 127.13.229.124:44203: connect: Connection refused (error 111)
I20260808 06:04:22.873569 5163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:33723,127.13.229.125:34679,127.13.229.124:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.229.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:22.874598 5163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:22.875576 5163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:22.883615 5172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.883632 5168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:22.883620 5169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:22.883805 5163 server_base.cc:1034] running on GCE node
I20260808 06:04:22.884462 5163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:22.885430 5163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:22.886595 5163 hybrid_clock.cc:648] HybridClock initialized: now 1786169062886579 us; error 25 us; skew 500 ppm
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:22.889286 5163 init.cc:377] Logged in from keytab as kudu/127.13.229.124@KRBTEST.COM (short username kudu)
I20260808 06:04:22.890182 5163 webserver.cc:460] Webserver started at http://127.13.229.124:43361/ using document root <none> and password file <none>
I20260808 06:04:22.890735 5163 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:22.890828 5163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:22.891073 5163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:22.893139 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data/instance:
uuid: "acd4ac7a71b84134b97d1d9851d0b112"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "202242739fd7c41969d4406da882d124"
server_key_iv: "f95fb881758cb9d10447da6f6bcabc8b"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.893702 5163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal/instance:
uuid: "acd4ac7a71b84134b97d1d9851d0b112"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "202242739fd7c41969d4406da882d124"
server_key_iv: "f95fb881758cb9d10447da6f6bcabc8b"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.897367 5163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 06:04:22.899703 5180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:22.900964 5163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:04:22.901111 5163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "acd4ac7a71b84134b97d1d9851d0b112"
format_stamp: "Formatted at 2026-08-08 06:04:22 on dist-test-slave-zkpd"
server_key: "202242739fd7c41969d4406da882d124"
server_key_iv: "f95fb881758cb9d10447da6f6bcabc8b"
server_key_version: "encryptionkey@0"
I20260808 06:04:22.901221 5163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:22.914491 5163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:22.917333 5163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:22.917532 5163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:22.924520 5163 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.124:44203
I20260808 06:04:22.924813 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.124:44203 every 8 connection(s)
I20260808 06:04:22.925534 5163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
I20260808 06:04:22.928331 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:22.932055 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.935819 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5163
I20260808 06:04:22.935974 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786168783448266-14229-0/minicluster-data/master-2/wal/instance
I20260808 06:04:22.936236 14229 external_mini_cluster.cc:1428] Setting key 0a086859b5fdee3343fe6a4782a8fb0e
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:22.946266 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.933896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.13.229.126:33723)
0808 06:04:22.934073 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.934078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.934842 (+ 764us) server_negotiation.cc:407] Connection header received
0808 06:04:22.935718 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.935723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.935783 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.935924 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.937433 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.938102 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.939348 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.939609 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.939706 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.942211 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.942214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.942216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.942241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.943876 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.944531 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.944534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.944535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.944652 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.945000 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.945004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.945006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.945253 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.945369 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.945979 (+ 610us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.946119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 06:04:22.947311 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.934263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:33723 (local address 127.0.0.1:57110)
0808 06:04:22.934690 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.934732 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.935515 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.936164 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.936170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.936571 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.937252 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.937275 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.938242 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.938245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.939220 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.939230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.939755 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.939759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.940019 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.940654 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.940671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.942057 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.944050 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.944062 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.944075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.944403 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.944755 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.944757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.944758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.944899 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.945364 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.945369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.945638 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.945890 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.946191 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260808 06:04:22.947449 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.936878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.13.229.124:44203)
0808 06:04:22.937263 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.937271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.937291 (+ 20us) server_negotiation.cc:407] Connection header received
0808 06:04:22.937437 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.937442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.937627 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.937794 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.938574 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.939352 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.940474 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.940810 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.940940 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.943512 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.943519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.943532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.943562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.945460 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.945929 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.945935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.945937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.946029 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.946346 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.946349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.946352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.946626 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.946818 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.947064 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.947175 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":161,"threads_started":1}
I20260808 06:04:22.948628 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.952469 5076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 3
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:22.957732 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.949124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:34679 (local address 127.0.0.1:45722)
0808 06:04:22.949511 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.949523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.949621 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.949855 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.949858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.950020 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.950103 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.950107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.950856 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.950860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.951363 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.951377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.951867 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.951871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.951998 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.952481 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.952497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.954040 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.956198 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.956201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.956203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.956441 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.956757 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.956759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.956760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.956828 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.957230 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.957233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.957331 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.957451 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.957574 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260808 06:04:22.957769 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.949254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.13.229.125:34679)
0808 06:04:22.949405 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.949410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.949586 (+ 176us) server_negotiation.cc:407] Connection header received
0808 06:04:22.949701 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.949704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.949755 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.949842 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.950221 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.950744 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.951494 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.951756 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.951867 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.954203 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.954207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.954209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.954238 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.956082 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.956555 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.956559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.956560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.956648 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.956925 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.956931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.956935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.957125 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.957239 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.957506 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.957628 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:22.958796 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:22.959478 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.949829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.13.229.126:33723)
0808 06:04:22.949942 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.949946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.949956 (+ 10us) server_negotiation.cc:407] Connection header received
0808 06:04:22.949993 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.949995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.950028 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.950111 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.950682 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.951148 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.952009 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.952325 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.952441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.955616 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.955620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.955623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.955649 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.957382 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.957894 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.957898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.957900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.957984 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.958324 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.958327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.958329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.958538 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.958749 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.959198 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.959316 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 06:04:22.961880 5154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } attempt: 3
I20260808 06:04:22.963106 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.952890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37258)
0808 06:04:22.953016 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.953032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.953111 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.953583 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.953586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.953783 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.953869 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.953879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.954877 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.954881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.955824 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.955835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.956564 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.956569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.956670 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.957554 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.957572 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.958970 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.961181 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.961184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.961186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.961459 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.961760 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.961764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.961766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.961854 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.962579 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.962583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.962688 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.962797 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.962947 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 06:04:22.963686 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.953249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.13.229.124:44203)
0808 06:04:22.953384 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.953388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.953401 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:04:22.953439 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.953442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.953478 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.953566 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.953996 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.954756 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.955995 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.956362 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.956545 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.959113 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.959118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.959122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.959147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.961077 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.961574 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.961577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.961578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.961678 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.961956 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.961961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.961962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.962206 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.962586 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.963428 (+ 842us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.963545 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
Aug 08 06:04:22 dist-test-slave-zkpd krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169062, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:22.968525 5076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: Bootstrap starting.
I20260808 06:04:22.971716 5076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:22.971796 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.960807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.13.229.125:34679)
0808 06:04:22.960920 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.960925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.960939 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:04:22.961047 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.961050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.961095 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.961182 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.962097 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.962859 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.963958 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.964292 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.964364 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.967750 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.967754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.967757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.967781 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.969566 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.970141 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.970148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.970152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.970251 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.970622 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.970626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.970629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.970889 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.971092 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.971494 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.971616 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:04:22.972117 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.962253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37276)
0808 06:04:22.962355 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.962381 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.962449 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.962973 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.962976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.963142 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.963258 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.963264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.964384 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.964388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.965142 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.965153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.965806 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.965810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.965911 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.966438 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.966451 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.968134 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.970499 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.970503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.970504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.970734 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.971055 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.971058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.971060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.971145 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.971592 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.971598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.971705 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.971795 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.971964 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 06:04:22.972852 5076 log.cc:824] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:22.973124 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.962473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.13.229.124:44203)
0808 06:04:22.962736 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.962746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.962760 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:04:22.962794 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.962797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.962838 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.963364 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.963490 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.964249 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.965284 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.965667 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.966229 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.968608 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.968619 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.968622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.968648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.970376 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.970856 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.970859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.970861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.970947 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.971255 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.971258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.971259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.971468 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.972551 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.972625 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.972967 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260808 06:04:22.974126 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.959268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:44203 (local address 127.0.0.1:37262)
0808 06:04:22.959482 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:22.959496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:22.960195 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:22.961342 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:22.961345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:22.961504 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:22.961607 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.961613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.963571 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.963574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.964386 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:22.964397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.965123 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.965126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:22.965211 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.965631 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:22.965649 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:22.968489 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:22.971541 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.971544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.971546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.971766 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.972603 (+ 837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:22.972606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:22.972607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:22.972705 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:22.973537 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:22.973540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:22.973640 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 06:04:22.973762 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 06:04:22.973937 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"spinlock_wait_cycles":2944}
I20260808 06:04:22.975564 5076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: No bootstrap required, opened a new log
I20260808 06:04:22.975755 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:22.976694 5154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: Bootstrap starting.
I20260808 06:04:22.977067 5050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cc38893cd3674522a4415ea0ca4b4871"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57112
I20260808 06:04:22.979311 5154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:22.979602 5076 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.979846 5076 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc38893cd3674522a4415ea0ca4b4871, State: Initialized, Role: FOLLOWER
I20260808 06:04:22.980058 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.959474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.13.229.124:44203)
0808 06:04:22.959831 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.959840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.959853 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:04:22.961168 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.961170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.961203 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.961334 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:22.961738 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.963460 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.964509 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:22.964886 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:22.965011 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:22.969178 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:22.969185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:22.969188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:22.969215 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:22.971440 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.972151 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.972154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.972155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.972494 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:22.972833 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:22.972836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:22.972838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:22.973391 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:22.973502 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:22.973833 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 06:04:22.974014 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260808 06:04:22.980189 5154 log.cc:824] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:22.980579 5076 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.981133 5244 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:22.981340 5244 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:22.981950 5076 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:22.982601 5154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: No bootstrap required, opened a new log
I20260808 06:04:22.982878 5235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: Bootstrap starting.
I20260808 06:04:22.985673 5235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:22.986517 5235 log.cc:824] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:22.986670 5154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.986917 5154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197f21662874459cae188a0d2673495b, State: Initialized, Role: FOLLOWER
W20260808 06:04:22.987109 5255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:22.987210 5255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:22.987260 5154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.987746 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:22.988015 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:22.988016 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 06:04:22.988135 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:22.988629 5235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: No bootstrap required, opened a new log
I20260808 06:04:22.989485 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.986636 5003 master_runner.cc:186] Error getting master registration for 127.13.229.125:34679: OK, instance_id { permanent_uuid: "197f21662874459cae188a0d2673495b" instance_seqno: 1786169062707426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:04:22.990154 5003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:04:22.990422 5050 leader_election.cc:288] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [CANDIDATE]: Term 1 election: Requested vote from peers 197f21662874459cae188a0d2673495b (127.13.229.125:34679), acd4ac7a71b84134b97d1d9851d0b112 (127.13.229.124:44203)
I20260808 06:04:22.992743 5235 raft_consensus.cc:348] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.992990 5235 raft_consensus.cc:729] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd4ac7a71b84134b97d1d9851d0b112, State: Initialized, Role: FOLLOWER
I20260808 06:04:22.993310 5235 consensus_queue.cc:260] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.993510 5154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:22.993552 5127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc38893cd3674522a4415ea0ca4b4871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "197f21662874459cae188a0d2673495b"
I20260808 06:04:22.993788 5127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:22.994976 5235 sys_catalog.cc:564] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:22.995038 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc38893cd3674522a4415ea0ca4b4871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "acd4ac7a71b84134b97d1d9851d0b112"
I20260808 06:04:22.995239 5208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:22.995312 5127 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc38893cd3674522a4415ea0ca4b4871 in term 1.
I20260808 06:04:22.996268 5024 leader_election.cc:302] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197f21662874459cae188a0d2673495b, cc38893cd3674522a4415ea0ca4b4871; no voters:
I20260808 06:04:22.996577 5244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:22.996809 5244 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [term 1 LEADER]: Becoming Leader. State: Replica: cc38893cd3674522a4415ea0ca4b4871, State: Running, Role: LEADER
I20260808 06:04:22.997174 5244 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } }
I20260808 06:04:22.999130 5244 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc38893cd3674522a4415ea0ca4b4871. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:22.999271 5244 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:23.001827 5208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc38893cd3674522a4415ea0ca4b4871 in term 1.
I20260808 06:04:23.002985 5263 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.003139 5263 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:23.006012 5293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:23.006971 5293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 06:04:23.010891 5289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:23.010983 5289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:23.011992 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.998155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.13.229.124:44203)
0808 06:04:22.998270 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.998275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.999752 (+ 1477us) server_negotiation.cc:407] Connection header received
0808 06:04:22.999975 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.999978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.000024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.000078 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.001202 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.001951 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.003879 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.004285 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.004387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:23.007303 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:23.007307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:23.007309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:23.007337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:23.009119 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.010467 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.010471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.010473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.010568 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.010878 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.010881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.010882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.011132 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:23.011281 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:23.011655 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 06:04:23.011783 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 06:04:23.013432 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.993118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.13.229.126:33723)
0808 06:04:22.993235 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.993240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:22.998762 (+ 5522us) server_negotiation.cc:407] Connection header received
0808 06:04:22.998969 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:22.998972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:22.999021 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:22.999073 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.000199 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.000997 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.003538 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.003874 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.003989 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:23.007359 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:23.007365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:23.007368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:23.007411 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:23.009710 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.010327 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.010331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.010332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.010435 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.010768 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.010773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.010776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.011021 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:23.011120 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:23.013154 (+ 2034us) server_negotiation.cc:299] Negotiation successful
0808 06:04:23.013289 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:04:23.017997 5208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [term 1 FOLLOWER]: Refusing update from remote peer cc38893cd3674522a4415ea0ca4b4871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:04:23.018735 5244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:04:23.019286 5294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:23.019425 5294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:23.019865 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:22.998848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.13.229.125:34679)
0808 06:04:22.998968 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:22.998975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.004639 (+ 5664us) server_negotiation.cc:407] Connection header received
0808 06:04:23.009264 (+ 4625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.009267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.009317 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.009387 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.011372 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.012122 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.014052 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:23.014372 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:23.014474 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:23.016191 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:23.016195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:23.016198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:23.016223 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:23.017960 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.018453 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.018457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.018458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.018553 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:23.018892 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:23.018896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:23.018897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:23.019120 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:23.019222 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:23.019600 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 06:04:23.019718 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 06:04:23.022360 5263 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc38893cd3674522a4415ea0ca4b4871. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.022485 5263 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:23.024164 5127 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [term 1 FOLLOWER]: Refusing update from remote peer cc38893cd3674522a4415ea0ca4b4871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:04:23.024748 5281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:04:23.029722 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc38893cd3674522a4415ea0ca4b4871. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.029838 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:23.032716 5281 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.032845 5281 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:23.033391 5244 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.033521 5244 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:23.034550 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.034659 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:23.034731 5301 mvcc.cc:205] Tried to move back new op lower bound from 7316148482113249280 to 7316148482040430592. Current Snapshot: MvccSnapshot[applied={T|T < 7316148482113249280}]
I20260808 06:04:23.035563 5293 catalog_manager.cc:1347] Generated new cluster ID: 1a9744c3a93046a8916015f0a653a617
I20260808 06:04:23.035625 5293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:23.035650 5263 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc38893cd3674522a4415ea0ca4b4871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc38893cd3674522a4415ea0ca4b4871" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 33723 } } peers { permanent_uuid: "197f21662874459cae188a0d2673495b" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 34679 } } peers { permanent_uuid: "acd4ac7a71b84134b97d1d9851d0b112" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 44203 } } }
I20260808 06:04:23.035784 5263 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:23.036563 5297 mvcc.cc:205] Tried to move back new op lower bound from 7316148482113249280 to 7316148482040430592. Current Snapshot: MvccSnapshot[applied={T|T < 7316148482113249280 or (T in {7316148482113249280})}]
I20260808 06:04:23.036757 5299 mvcc.cc:205] Tried to move back new op lower bound from 7316148482113249280 to 7316148482040430592. Current Snapshot: MvccSnapshot[applied={T|T < 7316148482113249280}]
I20260808 06:04:23.078065 5293 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:23.081575 5293 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:23.104470 5293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc38893cd3674522a4415ea0ca4b4871: Generated new TSK 0
I20260808 06:04:23.105543 5293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:04:23.139179 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.136775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.13.229.125:34679)
0808 06:04:23.136915 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.136920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.137989 (+ 1069us) server_negotiation.cc:407] Connection header received
0808 06:04:23.138196 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.138201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.138257 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.138311 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.139034 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45748: BlockingRecv error: recv got EOF from 127.0.0.1:45748 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 06:04:23.140916 5240 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.137640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.13.229.124:44203)
0808 06:04:23.137790 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.137795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.139625 (+ 1830us) server_negotiation.cc:407] Connection header received
0808 06:04:23.140071 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.140075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.140129 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.140182 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.140791 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37300: BlockingRecv error: recv got EOF from 127.0.0.1:37300 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 06:04:23.142829 5080 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.141458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.13.229.126:33723)
0808 06:04:23.141601 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.141606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.141922 (+ 316us) server_negotiation.cc:407] Connection header received
0808 06:04:23.142102 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.142105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.142152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.142202 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.142716 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57134: BlockingRecv error: recv got EOF from 127.0.0.1:57134 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 06:04:23.147531 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.145979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.13.229.125:34679)
0808 06:04:23.146084 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.146088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.146526 (+ 438us) server_negotiation.cc:407] Connection header received
0808 06:04:23.146767 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.146771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.146851 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.146910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.147421 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45750: BlockingRecv error: recv got EOF from 127.0.0.1:45750 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 06:04:23.152927 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.151092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.13.229.125:34679)
0808 06:04:23.151196 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.151200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.151751 (+ 551us) server_negotiation.cc:407] Connection header received
0808 06:04:23.152086 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.152090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.152144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.152200 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.152833 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45752: BlockingRecv error: recv got EOF from 127.0.0.1:45752 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 06:04:23.158385 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.156854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.13.229.125:34679)
0808 06:04:23.156962 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.156966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.157495 (+ 529us) server_negotiation.cc:407] Connection header received
0808 06:04:23.157687 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.157690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.157738 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.157789 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.158283 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45756: BlockingRecv error: recv got EOF from 127.0.0.1:45756 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 06:04:23.166061 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.164413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.13.229.125:34679)
0808 06:04:23.164553 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.164557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.165105 (+ 548us) server_negotiation.cc:407] Connection header received
0808 06:04:23.165313 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.165317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.165387 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.165443 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.165964 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45770: BlockingRecv error: recv got EOF from 127.0.0.1:45770 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260808 06:04:23.178047 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.176418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.13.229.125:34679)
0808 06:04:23.176523 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.176527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.177100 (+ 573us) server_negotiation.cc:407] Connection header received
0808 06:04:23.177285 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.177288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.177337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.177396 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.177922 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45780: BlockingRecv error: recv got EOF from 127.0.0.1:45780 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 06:04:23.197902 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.196518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.13.229.125:34679)
0808 06:04:23.196651 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.196658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.196948 (+ 290us) server_negotiation.cc:407] Connection header received
0808 06:04:23.197113 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.197116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.197173 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.197263 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.197774 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45782: BlockingRecv error: recv got EOF from 127.0.0.1:45782 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 06:04:23.233414 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.231996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.13.229.125:34679)
0808 06:04:23.232127 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.232132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.232465 (+ 333us) server_negotiation.cc:407] Connection header received
0808 06:04:23.232617 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.232621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.232683 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.232774 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.233289 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45792: BlockingRecv error: recv got EOF from 127.0.0.1:45792 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 06:04:23.301204 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.299973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.13.229.125:34679)
0808 06:04:23.300121 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.300125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.300308 (+ 183us) server_negotiation.cc:407] Connection header received
0808 06:04:23.300451 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.300454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.300506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.300589 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.301104 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45796: BlockingRecv error: recv got EOF from 127.0.0.1:45796 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 06:04:23.432566 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.431473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.13.229.125:34679)
0808 06:04:23.431658 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.431661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.431684 (+ 23us) server_negotiation.cc:407] Connection header received
0808 06:04:23.431838 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.431841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.431917 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.431993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.432433 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45810: BlockingRecv error: recv got EOF from 127.0.0.1:45810 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 06:04:23.691859 5160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:23.690764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.13.229.125:34679)
0808 06:04:23.690943 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:23.690947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:23.690985 (+ 38us) server_negotiation.cc:407] Connection header received
0808 06:04:23.691113 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:23.691116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:23.691166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:23.691241 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:23.691707 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45824: BlockingRecv error: recv got EOF from 127.0.0.1:45824 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 06:04:24.012426 5289 catalog_manager.cc:1259] Loaded cluster ID: 1a9744c3a93046a8916015f0a653a617
I20260808 06:04:24.012514 5289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: loading cluster ID for follower catalog manager: success
I20260808 06:04:24.014199 5289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: acquiring CA information for follower catalog manager: success
I20260808 06:04:24.015326 5289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 197f21662874459cae188a0d2673495b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:04:24.020551 5294 catalog_manager.cc:1259] Loaded cluster ID: 1a9744c3a93046a8916015f0a653a617
I20260808 06:04:24.020612 5294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: loading cluster ID for follower catalog manager: success
I20260808 06:04:24.022091 5294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: acquiring CA information for follower catalog manager: success
I20260808 06:04:24.023134 5294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P acd4ac7a71b84134b97d1d9851d0b112: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:04:24.209558 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:24.205975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.13.229.125:34679)
0808 06:04:24.206209 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:24.206213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:24.206254 (+ 41us) server_negotiation.cc:407] Connection header received
0808 06:04:24.206397 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:24.206400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:24.206532 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:24.206638 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:24.206874 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.207397 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.208236 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.208454 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.208534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:24.208802 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:24.209244 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:24.209337 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:24.209389 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 06:04:24.209435 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260808 06:04:24.215992 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:24.212695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.13.229.124:44203)
0808 06:04:24.212936 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:24.212940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:24.212953 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:04:24.213029 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:24.213031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:24.213135 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:24.213226 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:24.213418 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.213936 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.214673 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.214916 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.214988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:24.215229 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:24.215598 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:24.215682 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:24.215738 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 06:04:24.215784 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260808 06:04:24.223079 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:24.218756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.13.229.125:34679)
0808 06:04:24.218881 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:24.218888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:24.219302 (+ 414us) server_negotiation.cc:407] Connection header received
0808 06:04:24.219498 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:24.219501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:24.219560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:24.219637 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:24.219954 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.220736 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.221924 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.222150 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.222262 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:24.222592 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:24.222665 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:24.222743 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:24.222863 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 06:04:24.222918 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 06:04:24.223109 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:24.219101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.13.229.124:44203)
0808 06:04:24.219222 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:24.219227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:24.219586 (+ 359us) server_negotiation.cc:407] Connection header received
0808 06:04:24.219636 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:24.219639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:24.219680 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:24.219749 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:24.220083 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.220599 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.221878 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.222119 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.222232 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:24.222605 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:24.222679 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:24.222727 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:24.222911 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 06:04:24.222964 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 06:04:24.223979 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:24.218726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.13.229.126:33723)
0808 06:04:24.218962 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:24.218966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:24.219148 (+ 182us) server_negotiation.cc:407] Connection header received
0808 06:04:24.219404 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:24.219410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:24.219556 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:24.219674 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:24.220176 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.220952 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.222303 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:24.222660 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:24.222767 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:24.223200 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:24.223521 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:24.223617 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:24.223736 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 06:04:24.223811 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260808 06:04:24.226955 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5003
I20260808 06:04:24.238770 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5084
I20260808 06:04:24.246156 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5163
2026-08-08T06:04:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3189 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 06:04:24 dist-test-slave-zkpd krb5kdc[5395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 06:04:24 dist-test-slave-zkpd krb5kdc[5395](info): set up 2 sockets
Aug 08 06:04:24 dist-test-slave-zkpd krb5kdc[5395](info): commencing operation
krb5kdc: starting...
W20260808 06:04:25.317255 14229 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/test-user.keytab.
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T06:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T06:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.229.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
Entry for principal kudu/127.13.229.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab.
I20260808 06:04:25.442575 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34413
--webserver_interface=127.13.229.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 06:04:25.556900 5408 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:25.557163 5408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:25.557188 5408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:25.557224 5408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:25.561030 5408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:25.561075 5408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:25.561093 5408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:25.561110 5408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:25.561125 5408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:25.566139 5408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.126.keytab
--principal=kudu/127.13.229.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.126:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.229.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:25.567090 5408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:25.568118 5408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:25.576574 5417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:25.576565 5415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:25.576586 5414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:25.576839 5408 server_base.cc:1034] running on GCE node
I20260808 06:04:25.577558 5408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:25.578706 5408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:25.579948 5408 hybrid_clock.cc:648] HybridClock initialized: now 1786169065579912 us; error 39 us; skew 500 ppm
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:25.582665 5408 init.cc:377] Logged in from keytab as kudu/127.13.229.126@KRBTEST.COM (short username kudu)
I20260808 06:04:25.583515 5408 webserver.cc:460] Webserver started at http://127.13.229.126:34083/ using document root <none> and password file <none>
I20260808 06:04:25.584165 5408 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:25.584259 5408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:25.584501 5408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:25.586565 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data/instance:
uuid: "783feb6788154ec6988ee5357833adf0"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "b1b885309c450340b887374328f265da"
server_key_iv: "ce101c8d76689c0da3a4cbecd1ba8b1a"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.587147 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance:
uuid: "783feb6788154ec6988ee5357833adf0"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "b1b885309c450340b887374328f265da"
server_key_iv: "ce101c8d76689c0da3a4cbecd1ba8b1a"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.590894 5408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 06:04:25.593349 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:25.594537 5408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:25.594677 5408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal
uuid: "783feb6788154ec6988ee5357833adf0"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "b1b885309c450340b887374328f265da"
server_key_iv: "ce101c8d76689c0da3a4cbecd1ba8b1a"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.594789 5408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:25.606635 5408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:25.609592 5408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:25.609784 5408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:25.616676 5408 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.126:34413
I20260808 06:04:25.616961 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.126:34413 every 8 connection(s)
I20260808 06:04:25.617668 5408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/data/info.pb
I20260808 06:04:25.618852 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5408
I20260808 06:04:25.618991 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-0/wal/instance
I20260808 06:04:25.619288 14229 external_mini_cluster.cc:1428] Setting key 9b92af1ab66f296a92ad1d6902d84ff0
I20260808 06:04:25.620683 5482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:25.625240 5482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.229.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.125@KRBTEST.COM" created.
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:25.643590 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.627719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34413 (local address 127.0.0.1:50150)
0808 06:04:25.628540 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.628598 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:25.629939 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:25.630857 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:25.630866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:25.631513 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:25.631953 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.631978 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.633653 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.633656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.634395 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.634404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.635373 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.635379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.635536 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.636364 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:25.636389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:25.638211 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:25.640692 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.640701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.640709 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.640932 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.641318 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.641320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.641321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.641439 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.642109 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:25.642114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:25.642241 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 06:04:25.642389 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 06:04:25.642819 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":120,"spinlock_wait_cycles":37120,"thread_start_us":233,"threads_started":1}
I20260808 06:04:25.643595 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.627719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.13.229.126:34413)
0808 06:04:25.628587 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:25.628608 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:25.628719 (+ 111us) server_negotiation.cc:407] Connection header received
0808 06:04:25.630365 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:25.630378 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:25.630622 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:25.630812 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:25.632132 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.633517 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.634530 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.635195 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.635385 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.638375 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:25.638385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:25.638399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:25.638427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:25.640562 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.641068 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.641078 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.641080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.641230 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.641528 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.641531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.641532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.642001 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:25.642109 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:25.642471 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 06:04:25.642749 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":733,"thread_start_us":162,"threads_started":1}
I20260808 06:04:25.646024 5482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:25.646917 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.646563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44246)
0808 06:04:25.646738 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.646830 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 06:04:25.647122 5427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
I20260808 06:04:25.648141 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.647862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44252)
0808 06:04:25.648027 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.648080 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260808 06:04:25.648322 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:46541: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
Entry for principal kudu/127.13.229.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab.
I20260808 06:04:25.650640 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:46541
--webserver_interface=127.13.229.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:04:25.697688 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } attempt: 1
I20260808 06:04:25.698719 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.698358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44266)
0808 06:04:25.698519 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.698607 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 06:04:25.699517 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.699259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44276)
0808 06:04:25.699388 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.699448 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 06:04:25.699679 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:46541: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
W20260808 06:04:25.763868 5490 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:25.764137 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:25.764159 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:25.764195 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 06:04:25.767841 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } attempt: 2
W20260808 06:04:25.768206 5490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:25.768251 5490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:25.768270 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:25.768312 5490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:25.768344 5490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:25.768764 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.768438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44286)
0808 06:04:25.768605 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.768675 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 06:04:25.769527 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.769270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44298)
0808 06:04:25.769409 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.769465 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 06:04:25.769670 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.125:46541: Network error: Client connection negotiation failed: client connection to 127.13.229.125:46541: connect: Connection refused (error 111)
I20260808 06:04:25.774145 5490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.125.keytab
--principal=kudu/127.13.229.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.125:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.229.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:25.775084 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:25.776067 5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:25.784257 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:25.784327 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:25.784348 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:25.784622 5490 server_base.cc:1034] running on GCE node
I20260808 06:04:25.785281 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:25.786329 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:25.787495 5490 hybrid_clock.cc:648] HybridClock initialized: now 1786169065787467 us; error 39 us; skew 500 ppm
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:25.790277 5490 init.cc:377] Logged in from keytab as kudu/127.13.229.125@KRBTEST.COM (short username kudu)
I20260808 06:04:25.791189 5490 webserver.cc:460] Webserver started at http://127.13.229.125:35351/ using document root <none> and password file <none>
I20260808 06:04:25.791734 5490 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:25.791826 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:25.792093 5490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:25.794126 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data/instance:
uuid: "e52f7c9e88e041e5b527afbf30f48925"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "6818f3fef481d651d505463baf743841"
server_key_iv: "dfc7d1627a71ac6c234aa24287f08228"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.794699 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal/instance:
uuid: "e52f7c9e88e041e5b527afbf30f48925"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "6818f3fef481d651d505463baf743841"
server_key_iv: "dfc7d1627a71ac6c234aa24287f08228"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.798549 5490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 06:04:25.801028 5506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:25.802160 5490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 06:04:25.802300 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal
uuid: "e52f7c9e88e041e5b527afbf30f48925"
format_stamp: "Formatted at 2026-08-08 06:04:25 on dist-test-slave-zkpd"
server_key: "6818f3fef481d651d505463baf743841"
server_key_iv: "dfc7d1627a71ac6c234aa24287f08228"
server_key_version: "encryptionkey@0"
I20260808 06:04:25.802412 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:25.832230 5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:25.834940 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:25.835140 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:25.842270 5490 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.125:46541
I20260808 06:04:25.842588 5563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.125:46541 every 8 connection(s)
I20260808 06:04:25.843212 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/data/info.pb
I20260808 06:04:25.845890 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:25.847669 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5490
I20260808 06:04:25.847769 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-1/wal/instance
I20260808 06:04:25.848112 14229 external_mini_cluster.cc:1428] Setting key 4232d9d4deabfc7bff2f6c11855e126b
I20260808 06:04:25.849812 5564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.229.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.229.124@KRBTEST.COM" created.
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
I20260808 06:04:25.860816 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } attempt: 3
I20260808 06:04:25.866427 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.851660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.13.229.126:34413)
0808 06:04:25.851822 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:25.851827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:25.852787 (+ 960us) server_negotiation.cc:407] Connection header received
0808 06:04:25.854263 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:25.854265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:25.854318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:25.854393 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:25.856066 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.856592 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.858204 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.858447 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.858531 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.861247 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:25.861250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:25.861252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:25.861277 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:25.863146 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.863786 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.863789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.863791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.863876 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.864271 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.864274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.864276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.864467 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:25.864576 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:25.866019 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0808 06:04:25.866176 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 06:04:25.866442 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.852083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34413 (local address 127.0.0.1:50158)
0808 06:04:25.852617 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.852678 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:25.854002 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:25.854621 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:25.854634 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:25.855158 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:25.855855 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.855879 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.856737 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.856741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.858084 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.858095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.858585 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.858587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.858774 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.859293 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:25.859309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:25.861047 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:25.863264 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.863272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.863290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.863670 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.864009 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.864012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.864014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.864162 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.864576 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:25.864583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:25.864883 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 06:04:25.865085 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 06:04:25.865395 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:04:25.868168 5564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:25.872866 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.861260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44308)
0808 06:04:25.861427 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.861444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:25.861530 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:25.862863 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:25.862866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:25.863134 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:25.863269 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.863275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.864786 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.864789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.865555 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.865567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.866325 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.866329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.866432 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.866897 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:25.866922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:25.868454 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:25.871013 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.871018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.871020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.871254 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.871594 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.871597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.871599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.871695 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.872323 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:25.872329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:25.872533 (+ 204us) client_negotiation.cc:769] Sending connection context
0808 06:04:25.872642 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 06:04:25.872748 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 06:04:25.873086 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.861634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.13.229.125:46541)
0808 06:04:25.862368 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:25.862381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:25.862419 (+ 38us) server_negotiation.cc:407] Connection header received
0808 06:04:25.862466 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:25.862475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:25.862690 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:25.862839 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:25.863394 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.864659 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.865696 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.866183 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.867505 (+ 1322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.868608 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:25.868618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:25.868657 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:25.868687 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:25.870888 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.871382 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.871391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.871396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.871493 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.871790 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.871793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.871794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.872198 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:25.872323 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:25.872739 (+ 416us) server_negotiation.cc:299] Negotiation successful
0808 06:04:25.872871 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":614,"thread_start_us":128,"threads_started":1}
Aug 08 06:04:25 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:04:25.875301 5482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:25.876065 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.875775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34948)
0808 06:04:25.875954 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.876008 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 06:04:25.876854 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.876607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34962)
0808 06:04:25.876758 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.876800 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260808 06:04:25.877063 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
Entry for principal kudu/127.13.229.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab.
I20260808 06:04:25.878536 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.868726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44314)
0808 06:04:25.868844 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.868859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:25.868938 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:25.869401 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:25.869413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:25.869582 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:25.869675 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.869680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.870799 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.870806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.871343 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:25.871355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.871929 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.871932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:25.872048 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.872590 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:25.872603 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:25.874859 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:25.877256 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.877259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.877260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.877445 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.877712 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:25.877714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:25.877715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:25.877791 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:25.878126 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:25.878128 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:25.878199 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 06:04:25.878275 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 06:04:25.878406 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 06:04:25.878780 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.868786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.13.229.125:46541)
0808 06:04:25.869148 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:25.869159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:25.869173 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:04:25.869209 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:25.869214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:25.869264 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:25.869371 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:25.869801 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.870679 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.871463 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:25.871794 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:25.871930 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:25.875103 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:25.875107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:25.875108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:25.875130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:25.877160 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.877565 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.877568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.877569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.877644 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:25.877876 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:25.877878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:25.877879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:25.878063 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:25.878137 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:25.878358 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 06:04:25.878513 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":284,"thread_start_us":145,"threads_started":1}
I20260808 06:04:25.879423 14229 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNODW_f/build/debug/bin/kudu
/tmp/dist-test-taskNODW_f/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:35879
--webserver_interface=127.13.229.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.229.84:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 06:04:25.879406 5564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:25.880232 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.879839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34964)
0808 06:04:25.880053 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.880140 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":96}
W20260808 06:04:25.880455 5509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
I20260808 06:04:25.881309 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.881102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34972)
0808 06:04:25.881208 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.881256 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 06:04:25.881475 5564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
I20260808 06:04:25.908232 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 1
I20260808 06:04:25.909042 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.908753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34974)
0808 06:04:25.908906 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.908966 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 06:04:25.909782 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.909549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:34990)
0808 06:04:25.909671 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.909725 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 06:04:25.909917 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
I20260808 06:04:25.930819 5564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 1
I20260808 06:04:25.931717 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.931431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35002)
0808 06:04:25.931582 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.931643 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 06:04:25.932493 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.932267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35004)
0808 06:04:25.932376 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.932440 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:04:25.932621 5564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
I20260808 06:04:25.985073 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 2
I20260808 06:04:25.985900 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.985612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35020)
0808 06:04:25.985758 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.985826 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 06:04:25.986640 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:25.986428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35036)
0808 06:04:25.986538 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:25.986588 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 06:04:25.986771 5482 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
W20260808 06:04:25.995088 5573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 06:04:25.995384 5573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 06:04:25.995456 5573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 06:04:25.995537 5573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 06:04:25.999656 5573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 06:04:25.999727 5573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 06:04:25.999774 5573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 06:04:25.999817 5573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 06:04:25.999878 5573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 06:04:26.000772 5564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 2
I20260808 06:04:26.001484 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.001215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35048)
0808 06:04:26.001347 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.001415 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 06:04:26.002195 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.001978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35050)
0808 06:04:26.002087 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.002135 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 06:04:26.002353 5564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.229.124:35879: Network error: Client connection negotiation failed: client connection to 127.13.229.124:35879: connect: Connection refused (error 111)
I20260808 06:04:26.005307 5573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.229.84:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.229.126:34413,127.13.229.125:46541,127.13.229.124:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/krb5kdc/kudu_127.13.229.124.keytab
--principal=kudu/127.13.229.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.229.124:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.229.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 05:27:34 UTC on ccycloud.quasar-rquzrl.root.comops.site
build id 4574260
I20260808 06:04:26.006417 5573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 06:04:26.007445 5573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 06:04:26.015813 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:26.015813 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 06:04:26.015926 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 06:04:26.016032 5573 server_base.cc:1034] running on GCE node
I20260808 06:04:26.016815 5573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 06:04:26.017838 5573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 06:04:26.018997 5573 hybrid_clock.cc:648] HybridClock initialized: now 1786169066018962 us; error 45 us; skew 500 ppm
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 06:04:26.021852 5573 init.cc:377] Logged in from keytab as kudu/127.13.229.124@KRBTEST.COM (short username kudu)
I20260808 06:04:26.022670 5573 webserver.cc:460] Webserver started at http://127.13.229.124:42287/ using document root <none> and password file <none>
I20260808 06:04:26.023176 5573 fs_manager.cc:359] Metadata directory not provided
I20260808 06:04:26.023239 5573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 06:04:26.023398 5573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 06:04:26.025434 5573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data/instance:
uuid: "0f5ee22057b849979f15799ed2af23af"
format_stamp: "Formatted at 2026-08-08 06:04:26 on dist-test-slave-zkpd"
server_key: "010a90b89af7b4e354a519286ca238d6"
server_key_iv: "743ee704c7fa7fc256b2300f7efb075c"
server_key_version: "encryptionkey@0"
I20260808 06:04:26.025965 5573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal/instance:
uuid: "0f5ee22057b849979f15799ed2af23af"
format_stamp: "Formatted at 2026-08-08 06:04:26 on dist-test-slave-zkpd"
server_key: "010a90b89af7b4e354a519286ca238d6"
server_key_iv: "743ee704c7fa7fc256b2300f7efb075c"
server_key_version: "encryptionkey@0"
I20260808 06:04:26.029608 5573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 06:04:26.032085 5589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 06:04:26.033311 5573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 06:04:26.033458 5573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data,/tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal
uuid: "0f5ee22057b849979f15799ed2af23af"
format_stamp: "Formatted at 2026-08-08 06:04:26 on dist-test-slave-zkpd"
server_key: "010a90b89af7b4e354a519286ca238d6"
server_key_iv: "743ee704c7fa7fc256b2300f7efb075c"
server_key_version: "encryptionkey@0"
I20260808 06:04:26.033573 5573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 06:04:26.048354 5573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 06:04:26.051623 5573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 06:04:26.051828 5573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 06:04:26.058707 5573 rpc_server.cc:307] RPC server started. Bound to: 127.13.229.124:35879
I20260808 06:04:26.059021 5645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.229.124:35879 every 8 connection(s)
I20260808 06:04:26.059669 5573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/data/info.pb
I20260808 06:04:26.062439 5646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 06:04:26.066082 5646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:26.066711 14229 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNODW_f/build/debug/bin/kudu as pid 5573
I20260808 06:04:26.066833 14229 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNODW_f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786168783448266-14229-0/minicluster-data/master-2/wal/instance
I20260808 06:04:26.067140 14229 external_mini_cluster.cc:1428] Setting key 2b20ba92b0dd9ec97e8f3302468812fc
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:26.081241 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.068310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.13.229.126:34413)
0808 06:04:26.068435 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.068439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.069059 (+ 620us) server_negotiation.cc:407] Connection header received
0808 06:04:26.069855 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.069879 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.069928 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.070018 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.071460 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.072211 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.073693 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.074038 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.074129 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.076911 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.076916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.076918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.076950 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.078796 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.079325 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.079329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.079331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.079425 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.079858 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.079860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.079861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.080168 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.080262 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.080891 (+ 629us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.081041 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 06:04:26.082142 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.068435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.13.229.124:35879)
0808 06:04:26.068820 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.068837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.068881 (+ 44us) server_negotiation.cc:407] Connection header received
0808 06:04:26.069626 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.069644 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.069896 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.070057 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.070792 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.072014 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.073434 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.073923 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.074088 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.077389 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.077415 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.077431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.077461 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.079352 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.079959 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.079965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.079966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.080047 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.080447 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.080450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.080452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.080714 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.080900 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.081168 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.081290 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260808 06:04:26.082142 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.068435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.126:34413 (local address 127.0.0.1:50172)
0808 06:04:26.068905 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.068952 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:26.069644 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:26.070830 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:26.070834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:26.071134 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:26.071317 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.071326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.072358 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.072362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.073574 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.073585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.074194 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.074198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.074348 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.074862 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:26.074890 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:26.076726 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:26.078906 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.078915 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.078923 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.079224 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.079542 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.079546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.079547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.079749 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.080258 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:26.080264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:26.080584 (+ 320us) client_negotiation.cc:769] Sending connection context
0808 06:04:26.080795 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 06:04:26.081112 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"spinlock_wait_cycles":36224,"thread_start_us":165,"threads_started":1}
I20260808 06:04:26.083453 5646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:26.085924 5482 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 3
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.126@KRBTEST.COM
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.126@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:26.093506 5564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } attempt: 3
I20260808 06:04:26.093614 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.084126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.13.229.125:46541)
0808 06:04:26.084264 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.084268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.084283 (+ 15us) server_negotiation.cc:407] Connection header received
0808 06:04:26.084354 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.084357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.084410 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.084497 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.084953 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.085480 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.086523 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.086881 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.086963 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.089380 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.089384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.089387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.089418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.091281 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.091931 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.091936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.091938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.092041 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.092632 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.092635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.092636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.092816 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.092981 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.093314 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.093415 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 06:04:26.093659 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.084024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.125:46541 (local address 127.0.0.1:44316)
0808 06:04:26.084181 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.084196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:26.084268 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:26.084510 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:26.084513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:26.084680 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:26.084804 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.084815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.085590 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.085595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.086265 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.086276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.087119 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.087122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.087258 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.087788 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:26.087814 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:26.089236 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:26.091506 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.091509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.091511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.091774 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.092152 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.092154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.092155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.092233 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.092994 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:26.093001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:26.093124 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 06:04:26.093269 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 06:04:26.093417 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":24}
I20260808 06:04:26.095309 5646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } has no permanent_uuid. Determining permanent_uuid...
I20260808 06:04:26.096287 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.084609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.13.229.126:34413)
0808 06:04:26.084757 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.084760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.084773 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:04:26.084871 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.084873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.084910 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.085009 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.085640 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.086618 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.087448 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.087876 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.088065 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.091357 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.091361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.091364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.091391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.093583 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.094174 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.094178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.094180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.094277 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.094632 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.094639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.094644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.094908 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.095155 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.095956 (+ 801us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.096083 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 06:04:26.096741 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.086293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35054)
0808 06:04:26.086412 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.086428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:26.086515 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:26.086736 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:26.086739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:26.086920 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:26.087010 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.087017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.088042 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.088045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.088863 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.088880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.089699 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.089703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.089808 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.090273 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:26.090293 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:26.091826 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:26.094708 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.094711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.094713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.094962 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.095337 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.095340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.095342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.095455 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.096222 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:26.096236 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:26.096340 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 06:04:26.096442 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 06:04:26.096580 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 06:04:26.097398 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.086397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.13.229.124:35879)
0808 06:04:26.086502 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.086506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.086518 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:04:26.086581 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.086584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.086619 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.087114 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.087184 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.087933 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.089034 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.089532 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.090040 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.092019 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.092022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.092025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.092051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.094587 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.095095 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.095098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.095100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.095197 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.095585 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.095590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.095593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.095833 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.096932 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.097023 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.097188 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":25}
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.125@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
I20260808 06:04:26.101217 5482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0: Bootstrap starting.
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169066, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.229.124@KRBTEST.COM for kudu/127.13.229.124@KRBTEST.COM
Aug 08 06:04:26 dist-test-slave-zkpd krb5kdc[5395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786169065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.229.125@KRBTEST.COM
I20260808 06:04:26.104550 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.093913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35070)
0808 06:04:26.094007 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.094023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:26.094101 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:26.094751 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:26.094753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:26.094938 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:26.095031 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.095037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.096556 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.096559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.097373 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.097383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.098133 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.098136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.098220 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.098662 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:26.098676 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:26.100147 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:26.102761 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.102764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.102767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.103031 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.103354 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.103357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.103359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.103453 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.103990 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:26.103995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:26.104101 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 06:04:26.104195 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 06:04:26.104334 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 06:04:26.104614 5482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:26.105346 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.094182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.13.229.124:35879)
0808 06:04:26.094525 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.094535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.094551 (+ 16us) server_negotiation.cc:407] Connection header received
0808 06:04:26.094585 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.094587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.094623 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.095141 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.095226 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.096437 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.097652 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.098007 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.098419 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.100292 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.100298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.100304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.100327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.102630 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.103158 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.103164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.103167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.103250 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.103586 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.103594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.103598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.103820 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.104856 (+ 1036us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.104946 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.105093 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260808 06:04:26.105656 5482 log.cc:824] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:26.107573 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.095763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.229.124:35879 (local address 127.0.0.1:35076)
0808 06:04:26.096015 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 06:04:26.096031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 06:04:26.096103 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 06:04:26.096780 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 06:04:26.096784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 06:04:26.097185 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0808 06:04:26.097265 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.097271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.098782 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.098785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.099686 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 06:04:26.099698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.100440 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.100445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 06:04:26.100544 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.101486 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 06:04:26.101499 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 06:04:26.103209 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 06:04:26.105770 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.105773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.105774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.106368 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.106664 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 06:04:26.106666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 06:04:26.106668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 06:04:26.106730 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 06:04:26.107098 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 06:04:26.107100 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 06:04:26.107176 (+ 76us) client_negotiation.cc:769] Sending connection context
0808 06:04:26.107278 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 06:04:26.107405 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260808 06:04:26.108127 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.095982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.13.229.124:35879)
0808 06:04:26.096276 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.096284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.096296 (+ 12us) server_negotiation.cc:407] Connection header received
0808 06:04:26.096326 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.096328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.096571 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.096647 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.097387 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.098664 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.099878 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.100248 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.100331 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.103338 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.103341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.103343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.103364 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.105666 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.106496 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.106499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.106501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.106580 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.106814 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.106816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.106818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.106985 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.107066 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.107844 (+ 778us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.107973 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
I20260808 06:04:26.108529 5564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: Bootstrap starting.
I20260808 06:04:26.108865 5482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0: No bootstrap required, opened a new log
I20260808 06:04:26.109103 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.097631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.13.229.125:46541)
0808 06:04:26.097753 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.097758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.097963 (+ 205us) server_negotiation.cc:407] Connection header received
0808 06:04:26.098220 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.098223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.098273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.098328 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.099384 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.100164 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.101335 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.101665 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.101790 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.105042 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.105046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.105049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.105075 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.106958 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.107509 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.107513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.107515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.107609 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.108030 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.108034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.108036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.108307 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.108473 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.108865 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.108973 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 06:04:26.110875 5564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:26.111619 5564 log.cc:824] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:26.112727 14229 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 06:04:26.114063 5564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: No bootstrap required, opened a new log
I20260808 06:04:26.113965 5451 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "783feb6788154ec6988ee5357833adf0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50176
I20260808 06:04:26.114694 5646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: Bootstrap starting.
I20260808 06:04:26.115550 5482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.115808 5482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783feb6788154ec6988ee5357833adf0, State: Initialized, Role: FOLLOWER
I20260808 06:04:26.116262 5482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.117548 5646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: Neither blocks nor log segments found. Creating new log.
I20260808 06:04:26.117765 5482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:26.117853 5451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 06:04:26.117981 5451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:26.118046 5564 raft_consensus.cc:348] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.118252 5564 raft_consensus.cc:729] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52f7c9e88e041e5b527afbf30f48925, State: Initialized, Role: FOLLOWER
I20260808 06:04:26.118466 5646 log.cc:824] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: Log is configured to *not* fsync() on all Append() calls
I20260808 06:04:26.118539 5564 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.119272 5451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.119786 5564 sys_catalog.cc:564] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:26.120123 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.120931 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.121306 5451 leader_election.cc:288] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [CANDIDATE]: Term 1 election: Requested vote from peers e52f7c9e88e041e5b527afbf30f48925 (127.13.229.125:46541), 0f5ee22057b849979f15799ed2af23af (127.13.229.124:35879)
I20260808 06:04:26.121670 5536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "783feb6788154ec6988ee5357833adf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e52f7c9e88e041e5b527afbf30f48925"
I20260808 06:04:26.121932 5536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:26.123312 5536 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783feb6788154ec6988ee5357833adf0 in term 1.
I20260808 06:04:26.123728 5427 leader_election.cc:302] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783feb6788154ec6988ee5357833adf0, e52f7c9e88e041e5b527afbf30f48925; no voters:
I20260808 06:04:26.124189 5656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 06:04:26.124423 5656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [term 1 LEADER]: Becoming Leader. State: Replica: 783feb6788154ec6988ee5357833adf0, State: Running, Role: LEADER
I20260808 06:04:26.124744 5656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.126556 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783feb6788154ec6988ee5357833adf0. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.126667 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:26.127053 5661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.127189 5661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.127646 5408 master_runner.cc:186] Error getting master registration for 127.13.229.125:46541: OK, instance_id { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" instance_seqno: 1786169065840403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:04:26.132140 5408 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:04:26.132236 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 06:04:26.134239 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.123602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.13.229.126:34413)
0808 06:04:26.123948 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.123953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.123967 (+ 14us) server_negotiation.cc:407] Connection header received
0808 06:04:26.124222 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.124225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.124281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.124366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.125461 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.126181 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.127330 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.127642 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.127719 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.129686 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.129690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.129692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.129716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.131744 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.132582 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.132589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.132593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.132687 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.133042 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.133046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.133048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.133261 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.133419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.133977 (+ 558us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.134096 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260808 06:04:26.136725 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.123844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.13.229.125:46541)
0808 06:04:26.123974 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.123981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.125054 (+ 1073us) server_negotiation.cc:407] Connection header received
0808 06:04:26.125486 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.125492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.125542 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.125592 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.128213 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.128931 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.130059 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.130362 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.130472 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.133061 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.133065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.133067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.133091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.134865 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.135367 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.135371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.135372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.135455 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.135758 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.135762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.135763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.135990 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.136098 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.136461 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.136568 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 06:04:26.137859 5694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 06:04:26.137887 5646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: No bootstrap required, opened a new log
W20260808 06:04:26.137925 5694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:26.139132 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 06:04:26.141360 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:26.124487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.13.229.124:35879)
0808 06:04:26.124604 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.124608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.125256 (+ 648us) server_negotiation.cc:407] Connection header received
0808 06:04:26.125386 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.125390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.125448 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.125517 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.130085 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.130822 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.132025 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:26.132359 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:26.132471 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:26.137564 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 06:04:26.137568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 06:04:26.137571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 06:04:26.137597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 06:04:26.139404 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.139923 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.139926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.139928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.140019 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 06:04:26.140359 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 06:04:26.140363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 06:04:26.140364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 06:04:26.140579 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 06:04:26.140714 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:26.141065 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 06:04:26.141199 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 06:04:26.145536 5646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.146000 5646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5ee22057b849979f15799ed2af23af, State: Initialized, Role: FOLLOWER
I20260808 06:04:26.143311 5490 master_runner.cc:186] Error getting master registration for 127.13.229.124:35879: OK, instance_id { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" instance_seqno: 1786169066056801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 06:04:26.146492 5490 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 06:04:26.146749 5646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } }
I20260808 06:04:26.147508 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.147732 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.148476 5646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: configured and running, proceeding with master startup.
I20260808 06:04:26.148571 5618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "783feb6788154ec6988ee5357833adf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0f5ee22057b849979f15799ed2af23af"
I20260808 06:04:26.148862 5618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [term 0 FOLLOWER]: Advancing to term 1
I20260808 06:04:26.150319 5618 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783feb6788154ec6988ee5357833adf0 in term 1.
I20260808 06:04:26.155859 5536 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [term 1 FOLLOWER]: Refusing update from remote peer 783feb6788154ec6988ee5357833adf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:04:26.156325 5618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [term 1 FOLLOWER]: Refusing update from remote peer 783feb6788154ec6988ee5357833adf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 06:04:26.156991 5689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 06:04:26.157143 5656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 06:04:26.160648 5711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 06:04:26.160763 5711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 06:04:26.161612 5661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783feb6788154ec6988ee5357833adf0. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.161723 5661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.162900 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783feb6788154ec6988ee5357833adf0. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.163002 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.163779 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.163991 5661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.164089 5661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.164251 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:26.165189 5693 catalog_manager.cc:1347] Generated new cluster ID: bd0be5160aa04c69a40c96a5517b00bc
I20260808 06:04:26.165246 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 06:04:26.178478 5714 mvcc.cc:205] Tried to move back new op lower bound from 7316148494965338112 to 7316148494850600960. Current Snapshot: MvccSnapshot[applied={T|T < 7316148494965338112}]
I20260808 06:04:26.179169 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.179311 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0 [sys.catalog]: This master's current role is: LEADER
I20260808 06:04:26.180636 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "783feb6788154ec6988ee5357833adf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783feb6788154ec6988ee5357833adf0" member_type: VOTER last_known_addr { host: "127.13.229.126" port: 34413 } } peers { permanent_uuid: "e52f7c9e88e041e5b527afbf30f48925" member_type: VOTER last_known_addr { host: "127.13.229.125" port: 46541 } } peers { permanent_uuid: "0f5ee22057b849979f15799ed2af23af" member_type: VOTER last_known_addr { host: "127.13.229.124" port: 35879 } } }
I20260808 06:04:26.180751 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af [sys.catalog]: This master's current role is: FOLLOWER
I20260808 06:04:26.180873 5716 mvcc.cc:205] Tried to move back new op lower bound from 7316148494965338112 to 7316148494850600960. Current Snapshot: MvccSnapshot[applied={T|T < 7316148494965338112}]
I20260808 06:04:26.187700 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 06:04:26.188944 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260808 06:04:26.203385 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 783feb6788154ec6988ee5357833adf0: Generated new TSK 0
I20260808 06:04:26.204825 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 06:04:26.244302 5485 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.241427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.13.229.126:34413)
0808 06:04:26.241556 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.241561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.243101 (+ 1540us) server_negotiation.cc:407] Connection header received
0808 06:04:26.243311 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.243314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.243367 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.243421 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.244150 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50198: BlockingRecv error: recv got EOF from 127.0.0.1:50198 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 06:04:26.245829 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.242160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.13.229.125:46541)
0808 06:04:26.242275 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.242279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.244778 (+ 2499us) server_negotiation.cc:407] Connection header received
0808 06:04:26.245003 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.245007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.245056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.245107 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.245668 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44340: BlockingRecv error: recv got EOF from 127.0.0.1:44340 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 06:04:26.247484 5652 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.242808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.13.229.124:35879)
0808 06:04:26.242924 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.242930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.246335 (+ 3405us) server_negotiation.cc:407] Connection header received
0808 06:04:26.246749 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.246766 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.246818 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.246872 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.247371 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 06:04:26.251870 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.250434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.13.229.125:46541)
0808 06:04:26.250528 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.250531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.250963 (+ 432us) server_negotiation.cc:407] Connection header received
0808 06:04:26.251167 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.251170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.251241 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.251292 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.251736 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44346: BlockingRecv error: recv got EOF from 127.0.0.1:44346 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 06:04:26.257107 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.255722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.13.229.125:46541)
0808 06:04:26.255837 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.255841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.256308 (+ 467us) server_negotiation.cc:407] Connection header received
0808 06:04:26.256494 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.256497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.256540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.256587 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.257024 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44356: BlockingRecv error: recv got EOF from 127.0.0.1:44356 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 06:04:26.262418 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.261007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.13.229.125:46541)
0808 06:04:26.261097 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.261101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.261616 (+ 515us) server_negotiation.cc:407] Connection header received
0808 06:04:26.261804 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.261807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.261851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.261898 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.262335 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44370: BlockingRecv error: recv got EOF from 127.0.0.1:44370 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 06:04:26.269654 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.268246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.13.229.125:46541)
0808 06:04:26.268366 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.268369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.268859 (+ 490us) server_negotiation.cc:407] Connection header received
0808 06:04:26.269027 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.269031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.269073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.269119 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.269568 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44382: BlockingRecv error: recv got EOF from 127.0.0.1:44382 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 06:04:26.281874 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.280031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.13.229.125:46541)
0808 06:04:26.280166 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.280171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.280826 (+ 655us) server_negotiation.cc:407] Connection header received
0808 06:04:26.281064 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.281067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.281120 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.281182 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.281740 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44398: BlockingRecv error: recv got EOF from 127.0.0.1:44398 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 06:04:26.303318 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.301172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.13.229.125:46541)
0808 06:04:26.301321 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.301326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.302076 (+ 750us) server_negotiation.cc:407] Connection header received
0808 06:04:26.302350 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.302354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.302422 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.302486 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.303180 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44406: BlockingRecv error: recv got EOF from 127.0.0.1:44406 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 06:04:26.339845 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.338015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.13.229.125:46541)
0808 06:04:26.338139 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.338144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.338801 (+ 657us) server_negotiation.cc:407] Connection header received
0808 06:04:26.339015 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.339019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.339072 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.339123 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.339707 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44422: BlockingRecv error: recv got EOF from 127.0.0.1:44422 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 06:04:26.407953 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.406661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.13.229.125:46541)
0808 06:04:26.406822 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.406828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.407044 (+ 216us) server_negotiation.cc:407] Connection header received
0808 06:04:26.407187 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.407189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.407241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.407318 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.407771 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44424: BlockingRecv error: recv got EOF from 127.0.0.1:44424 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 06:04:26.539422 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.538165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.13.229.125:46541)
0808 06:04:26.538321 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.538326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.538470 (+ 144us) server_negotiation.cc:407] Connection header received
0808 06:04:26.538599 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.538602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.538656 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.538734 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.539252 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44426: BlockingRecv error: recv got EOF from 127.0.0.1:44426 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 06:04:26.798940 5571 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 06:04:26.797817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.13.229.125:46541)
0808 06:04:26.797977 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:26.797980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:26.797998 (+ 18us) server_negotiation.cc:407] Connection header received
0808 06:04:26.798128 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:26.798131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:26.798182 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:26.798257 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 06:04:26.798754 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44432: BlockingRecv error: recv got EOF from 127.0.0.1:44432 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 06:04:27.139294 5694 catalog_manager.cc:1259] Loaded cluster ID: bd0be5160aa04c69a40c96a5517b00bc
I20260808 06:04:27.139394 5694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: loading cluster ID for follower catalog manager: success
I20260808 06:04:27.141242 5694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: acquiring CA information for follower catalog manager: success
I20260808 06:04:27.142409 5694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e52f7c9e88e041e5b527afbf30f48925: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:04:27.162209 5711 catalog_manager.cc:1259] Loaded cluster ID: bd0be5160aa04c69a40c96a5517b00bc
I20260808 06:04:27.162351 5711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: loading cluster ID for follower catalog manager: success
I20260808 06:04:27.164243 5711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: acquiring CA information for follower catalog manager: success
I20260808 06:04:27.165427 5711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f5ee22057b849979f15799ed2af23af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 06:04:27.317507 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:27.313336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.13.229.125:46541)
0808 06:04:27.313645 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:27.313651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:27.313668 (+ 17us) server_negotiation.cc:407] Connection header received
0808 06:04:27.313835 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:27.313839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:27.314016 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:27.314136 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:27.314407 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.314974 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.315880 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.316152 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.316266 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:27.316606 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:27.317075 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:27.317216 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:27.317253 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 06:04:27.317313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260808 06:04:27.324852 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:27.321167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.13.229.124:35879)
0808 06:04:27.321423 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:27.321427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:27.321440 (+ 13us) server_negotiation.cc:407] Connection header received
0808 06:04:27.321535 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:27.321537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:27.321661 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:27.321816 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:27.322055 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.322577 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.323412 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.323625 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.323701 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:27.323995 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:27.324438 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:27.324533 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:27.324598 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 06:04:27.324644 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260808 06:04:27.332682 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:27.328756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.13.229.124:35879)
0808 06:04:27.328872 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:27.328877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:27.329025 (+ 148us) server_negotiation.cc:407] Connection header received
0808 06:04:27.329146 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:27.329150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:27.329201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:27.329277 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:27.329575 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.330317 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331492 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.331731 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331841 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:27.332201 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:27.332274 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:27.332365 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:27.332459 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 06:04:27.332522 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 06:04:27.332789 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:27.328350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.13.229.125:46541)
0808 06:04:27.328480 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:27.328483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:27.328767 (+ 284us) server_negotiation.cc:407] Connection header received
0808 06:04:27.329016 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:27.329019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:27.329072 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:27.329177 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:27.329507 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.330056 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331325 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.331668 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331778 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:27.332212 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:27.332299 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:27.332350 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:27.332580 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 06:04:27.332658 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 06:04:27.333015 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 06:04:27.328007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.13.229.126:34413)
0808 06:04:27.328260 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 06:04:27.328266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 06:04:27.328287 (+ 21us) server_negotiation.cc:407] Connection header received
0808 06:04:27.328449 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 06:04:27.328455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 06:04:27.328679 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 06:04:27.328820 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 06:04:27.329234 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.330134 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331119 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 06:04:27.331433 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 06:04:27.331543 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 06:04:27.332032 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 06:04:27.332400 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 06:04:27.332495 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 06:04:27.332787 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 06:04:27.332864 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260808 06:04:27.336429 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5408
I20260808 06:04:27.346607 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5490
I20260808 06:04:27.354900 14229 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNODW_f/build/debug/bin/kudu with pid 5573
2026-08-08T06:04:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3109 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6299 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283897 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST